RFC 9696
Routing in Fat Trees (RIFT) Applicability and Operational Considerations
RFC 9696: Routing in Fat Trees (RIFT) Applicability and Operational Considerations 🦋
okay bestie, this one is all about RIFT (Routing in Fat Trees) and when you should actually use it 💅 It's an Informational RFC, so it's a guide, not a rulebook. It's from April 2025 and it's an ISSN 2070-1721 doc.
The authors are Y. Wei, Ed. (ZTE Corporation), Z. Zhang (ZTE Corporation), D. Afanasiev (Yandex), P. Thubert (Individual), and T. Przygienda (Juniper Networks). The title block doesn't list anything this RFC obsoletes or updates 🌸
Abstract
okay so the abstract says this doc looks at the properties, applicability, and operational considerations of Routing in Fat Trees (RIFT) across different network scenarios. The goal is a rough guide to deploying RIFT to make routing easier in Clos topologies and their variations ✨
Status of This Memo
This document is not an Internet Standards Track specification; it is published for informational purposes.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are candidates for any level of Internet Standard; see Section 2 of RFC 7841.
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc9696 🎀
Copyright Notice
Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. 💜
Table of Contents
1. Introduction
2. Terminology
3. Problem Statement of Routing in Modern IP Fabric Fat Tree
Networks
4. Applicability of RIFT to Clos IP Fabrics
4.1. Overview of RIFT
4.2. Applicable Topologies
4.2.1. Horizontal Links
4.2.2. Vertical Shortcuts
4.2.3. Generalizing to Any Directed Acyclic Graph
4.2.4. Reachability of Internal Nodes in the Fabric
4.3. Use Cases
4.3.1. Data Center Topologies
4.3.2. Metro Networks
4.3.3. Building Cabling
4.3.4. Internal Router Switching Fabrics
4.3.5. CloudCO
5. Operational Considerations
5.1. South Reflection
5.2. Suboptimal Routing on Link Failures
5.3. Black-Holing on Link Failures
5.4. Zero Touch Provisioning (ZTP)
5.5. Miscabling
5.5.1. Miscabling Examples
5.5.2. Miscabling Considerations
5.6. Multicast and Broadcast Implementations
5.7. Positive vs. Negative Disaggregation
5.8. Mobile Edge and Anycast
5.9. IPv4 over IPv6
5.10. In-Band Reachability of Nodes
5.11. Dual-Homing Servers
5.12. Fabric with a Controller
5.12.1. Controller Attached to ToFs
5.12.2. Controller Attached to Leaf
5.13. Internet Connectivity Within Underlay
5.13.1. Internet Default on the Leaf
5.13.2. Internet Default on the ToFs
5.14. Subnet Mismatch and Address Families
5.15. Anycast Considerations
5.16. IoT Applicability
5.17. Key Management
5.18. TTL/Hop Limit of 1 vs. 255 on LIEs/TIEs
6. Security Considerations
7. IANA Considerations
8. References
8.1. Normative References
8.2. Informative References
Acknowledgments
Contributors
Authors' Addresses1. Introduction ✨
okay so this doc talks about the properties and applicability of "RIFT: Routing in Fat Trees" [RFC9692] across different deployment scenarios. It also shows how simple the tech is to operate compared to classic routing solutions 💖 It covers special considerations for using RIFT with or without overlays and/or controllers, and how RIFT spots miscablings and reroutes around node and link failures 🧚
2. Terminology 🌈
this doc uses the terminology defined in [RFC9692]. The most-used terms and their definitions from that doc are listed here, so let's go 👑
Clos / Fat Tree: this doc uses "Clos" and "Fat Tree" interchangeably, and both always mean a folded spine-and-leaf topology. There can be multiple Points of Delivery (PoDs) and one or multiple Top of Fabric (ToF) planes. Some modifications, like leaf-2-leaf shortcuts and multiple level shortcuts, are possible and get covered later in the doc 🍓
Crossbar: the physical arrangement of ports in a switching matrix. It doesn't imply any scheduling or buffering discipline 🫶
Directed Acyclic Graph (DAG): a finite directed graph with no directed cycles (loops). If the links in a Clos are all treated as directed toward the top or all toward the bottom, each of those two graphs is a DAG 🦋
Disaggregation: the process where a node decides to advertise more specific prefixes southwards, either positively to attract the matching traffic or negatively to repel it. Disaggregation exists to prevent traffic loss and suboptimal routing to those more specific prefixes 💅
Leaf: a node without southbound adjacencies. Level 0 implies a leaf in RIFT, but a leaf doesn't have to be level 0 🌸
LIE: short for "Link Information Element". LIEs get exchanged on all of the system's RIFT links to form ThreeWay adjacencies, and they carry the info used for RIFT Zero Touch Provisioning (ZTP) of levels ✨
South Reflection: often shortened to just "reflection". South Reflection is a mechanism where South Node TIEs get "reflected" from the level south back up north. This lets nodes in the same level that have no East-West links still learn about each other's node Topology Information Elements (TIEs) 🪞
Spine: any nodes north of leaves and south of ToF nodes. A PoD can have multiple layers of spines 💜
TIE: short for "Topology Information Element". TIEs get exchanged between RIFT nodes to describe parts of a network, like links and address prefixes. A TIE always has a direction and a type. North TIEs (sometimes abbreviated as N-TIEs) are used for the northbound representation, and South-TIEs (sometimes abbreviated as S-TIEs) are used for the southbound equivalent. TIEs come in different types, such as node and prefix TIEs 🎀
3. Problem Statement of Routing in Modern IP Fabric Fat Tree Networks 🌷
Clos [CLOS] topologies (commonly called a Fat Tree/network in modern IP fabric considerations, as a similar term for the original definition of the term Fat Tree [FATTREE]) have become a huge deal in networking today. This is mainly because of the big shift toward centralized, data-center-based architectures that deliver most of the computation and storage services 😭
Current routing protocols were built for networks with irregular topologies, isotropic properties, and a low degree of connectivity. When they get applied to Fat Tree topologies, a few things go wrong:
- They tend to need extensive configuration or provisioning during initialization, and also when nodes get added to or removed from the fabric 🙈
- For link-state routing protocols, all nodes, including spine-and-leaf nodes, learn the entire network topology and routing information. That's actually not needed on the leaf nodes during normal operation 💀 They flood significant amounts of duplicate link-state information between spine-and-leaf nodes during topology updates and convergence events, which eats extra CPU and link bandwidth. This can hurt the stability and scalability of the fabric, make it less reactive to failures, and block the use of cheaper hardware at the lower levels (i.e., spine-and-leaf nodes) 😤
4. Applicability of RIFT to Clos IP Fabrics 🧚♀️
okay so the rest of this doc assumes you already know the terms and concepts from the Open Shortest Path First (OSPF) [RFC2328], OSPF for IPv6 [RFC5340], and Intermediate System to Intermediate System (IS-IS) [ISO10589-Second-Edition] link-state protocols. [RFC9692] outlines the requirements for routing in IP fabrics and the RIFT protocol concepts 🔥
4.1. Overview of RIFT ✨
okay so RIFT is a dynamic routing protocol that's been custom-tailored for Clos, Fat Tree, and other anisotropic topologies 💅 and the core vibe of the whole thing is that it's sensitive to the shape of the fabric. it's anisotropic, bestie. 🦋 it's giving "I know which way is up" energy. when RIFT is "pointing north" it acts like a link-state protocol: it advertises southward routes to northward peers (parents) through flooding and database synchronization 🌸. when it's "pointing south" it works hop-by-hop like a distance-vector protocol, usually advertising a fabric default route toward the ToF (aka superspine) to southward peers (children) 🧚
the fabric default is typically the default route described in Section 6.3.8 ("Southbound Default Route Origination") of [RFC9692]. the ToF nodes may alternatively originate more specific prefixes (P') southbound instead of the default route 🎀 but in that case, all addresses carried within the RIFT domain must be contained within P', and a leaf that acts as a gateway to the Internet is allowed to advertise the default route instead 🌈
RIFT floods flat link-state info northbound only, so that each level gets the full topology of the levels south of it ✨ that info is never flooded East-West or back south again, so a top tier node ends up with a full set of prefixes from the Shortest Path First (SPF) calculation. iconic 👑
in the southbound direction the protocol acts like a "fully summarizing, unidirectional" path-vector protocol or, rather, a distance-vector with implicit split horizon 🍓 routing info (normally just the default route) propagates one hop south and then gets "re-advertised" by nodes at the next lower level 💜
+---------------+ +----------------+
| ToF | | ToF | LEVEL 2
+ ++------+--+--+-+ ++-+--+----+-----+
| | | | | | | | | ^
+ | | | +-------------------------+ |
Distance- | +-------------------+ | | | | |
Vector | | | | | | | | +
South | | | | +--------+ | | | Link-State
+ | | | | | | | | Flooding
| | | +----------------+ | | | North
v | | | | | | | | +
++---+-+ +------+ +-+----+ ++----++ |
|SPINE | |SPINE | | SPINE| | SPINE| | LEVEL 1
+ ++----++ ++---+-+ +-+--+-+ ++----++ |
+ | | | | | | | | | ^ N
Distance- | +-------+ | | +--------+ | | | E
Vector | | | | | | | | | +------>
South | +-------+ | | | +------+ | | | |
+ | | | | | | | | | +
v ++--++ +-+-++ ++--++ ++--++ +
|LEAF| |LEAF| |LEAF| |LEAF| LEVEL 0
+----+ +----+ +----+ +----+
Figure 1: RIFT Overviewa spine node only has the info it needs for its level 🦋 which is all destinations south of the node based on SPF calculation, the default route, and potentially disaggregated routes
RIFT combines the best parts of both link-state and distance-vector protocols, and it's honestly so main character 🌟:
- Fastest possible convergence 🏃♀️
- Automatic detection of topology 🔍
- Minimal routes/information on Top-of-Rack (ToR) switches, aka leaf nodes 🍃
- High degree of ECMP 🛣️
- Fast decommissioning of nodes 🧹
- Maximum propagation speed with flexible prefixes in an update 💨
there are two types of link-state databases: "north representation" North Topology Information Elements (N-TIEs) and "south representation" South Topology Information Elements (S-TIEs) 📚 the N-TIEs contain a link-state topology description of lower levels, and the S-TIEs simply carry default and disaggregated routes for the lower levels 💖
RIFT also gets rid of the major disadvantages of link-state and distance-vector protocols with these:
- Reduced and balanced flooding 🌊
- Level-constrained automatic neighbor discovery 🫶
to do all this, RIFT builds on the art of IGPs like OSPF, IS-IS, Mobile Ad Hoc Network (MANET), and Internet of Things (IoT) 🧠 and it adds some unique features:
- Automatic (positive or negative) route disaggregation of northward routes upon fallen leaves 🍂
- Recursive operation in the case of negative route disaggregation 🔁
- Anisotropic routing that extends a principle seen in the Routing Protocol for Low-Power and Lossy Networks (RPL) [RFC6550] to wide superspines 🌐
- Optimal flooding reduction that derives from the concept of a "multipoint relay" (MPR) found in Optimized Link State Routing (OLSR) [RFC3626], balancing the flooding load over northbound links and nodes ⚖️
more advantages that are unique to RIFT are listed below, and the details live in RIFT [RFC9692] 📖
- True ZTP 🪄
- Minimal blast radius on failures 💥
- Can utilize all paths through fabric without looping 🛤️
- Simple leaf implementation that can scale down to servers 🖥️
- Key-value store 🗝️
- Horizontal links used for protection only 🛡️
4.2. Applicable Topologies 🗺️
albeit RIFT is specified primarily for "proper" Clos or Fat Tree topologies, the protocol natively supports Points of Delivery (PoD) concepts 🏘️ which, strictly speaking, aren't found in the original Clos concept
further, the specification explains and supports operations of multi-plane Clos variants, where the protocol recommends the use of inter-plane rings at the ToF level 💍 to allow reconciliation of the topology view of different planes, which makes Negative Disaggregation viable when there are failures within a plane. these observations hold not only for RIFT but also for the generic case of dynamic routing on Clos variants with multiple planes and failures in bisectional bandwidth, especially on the leaves 🫠
4.2.1. Horizontal Links ↔️
RIFT isn't limited to pure Clos divided into PoDs and multi-planes, it also supports horizontal (East-West) links below the ToF level ✨ those links are used only for last-resort northbound forwarding, when a spine loses all its northbound links or can't compute a default route through them 🆘
a full-mesh connectivity between nodes on the same level can be deployed, which lets North SPF (N-SPF) provide for any node that loses all its northbound adjacencies (as long as any of the other nodes in the level are northbound connected) and still participate in northbound forwarding 🙌
note that a "ring" of horizontal links at any level below ToF does not provide a "ring-based protection" scheme, since the SPF computation would have to deal with breaking "loops", and that's an application RIFT just isn't intended for 🙅♀️
4.2.2. Vertical Shortcuts ⚡
through relaxations of the specified adjacency forming rules, RIFT implementations can be extended to support vertical "shortcuts" 🪜 the RIFT spec itself doesn't give the exact details, since the resulting solution suffers from either a much larger blast radius with increased flooding volumes, or bow tie problems in the case of maximum aggregation routing 🎀
4.2.3. Generalizing to Any Directed Acyclic Graph 🕸️
RIFT is an anisotropic routing protocol, meaning it has a sense of direction (northbound, southbound, and East-West) and operates differently depending on the direction 🧭
since a DAG provides a sense of north (the direction of the DAG) and south (the reverse), it can be used to apply RIFT. an edge in the DAG that has only incoming vertices is a ToF node 👑
there are a number of caveats though, bestie 🫣
- The DAG structure must exist before RIFT starts, so there's a need for a companion protocol to establish the logical DAG structure 🏗️
- A generic DAG doesn't have a sense of East and West. The operation specified for East-West links and the southbound reflection between nodes is not applicable. Also, ZTP will derive a sense of depth that eliminates some links. Variations of ZTP could be derived to meet specific objectives, e.g., making it so that most routers have at least two parents to reach the ToF 🧩
- RIFT applies to any Destination-Oriented DAG (DODAG) where there's only one ToF node and the problem of disaggregation doesn't exist. In that case, RIFT operates very much like RPL [RFC6550], but uses link-state information for southbound routes (downwards in RPL's terms). For an arbitrary DAG with multiple destinations (ToFs), the way disaggregation happens has to be considered 🌿
- Positive Disaggregation expects that most of the ToF nodes reach most of the leaves, so disaggregation is the exception rather than the rule. When this is no longer true, it makes sense to turn off disaggregation and route between the ToF nodes over a ring, a full mesh, a transit network, or a form of area zero. Then again, this operation is similar to RPL operating as a single DODAG with a virtual root 🌀
- In order to aggregate and disaggregate routes, RIFT requires that all the ToF nodes share full knowledge of the prefixes in the fabric. This can be achieved with a ring as suggested by RIFT [RFC9692], by some preconfiguration, or by synchronizing with a common repository where all the active prefixes are registered 🗄️
4.2.4. Reachability of Internal Nodes in the Fabric 📍
RIFT does not require that nodes have reachable addresses in the fabric, though it is clearly desirable for operational purposes 🧾 under normal operating conditions, this can be easily achieved by injecting the node's loopback address into Prefix North TIEs and Prefix South TIEs, or through other implementation-specific mechanisms 🔧
special considerations arise when a node loses all northbound adjacencies but is not at the top of the fabric 😬 if a spine node loses all northbound links, the spine node doesn't advertise a default route. But if the level of the spine node is auto-determined by ZTP, it will "fall down" as depicted in Figure 8 🪂
4.3. Use Cases 🎀
4.3.1. Data Center Topologies 🏢
4.3.1.1. Data Center Fabrics 💻
RIFT is suited for applying underlay routing in data center (DC) IP fabrics, and the vast majority of these fabrics are Clos architectures (and will be for the foreseeable future) 🌈 it significantly simplifies operation and deployment of such fabrics, as described in Section 5, compared to extensive proprietary provisioning and operational solutions 💅
Okay that's the first chunk of the RIFT deep dive done, bestie! 🫶 Figure 8 is coming up, so keep your eyes peeled for the next bit. 🦋✨
4.3.1.2. Adaptations to Other Proposed Data Center Topologies 🎀
. +-----+ +-----+
. | | | |
.+-+ S0 | | S1 |
.| ++---++ ++---++
.| | | | |
.| | +------------+ |
.| | | +------------+ |
.| | | | |
.| ++-+--+ +--+-++
.| | | | |
.| | A0 | | A1 |
.| +-+--++ ++---++
.| | | | |
.| | +------------+ |
.| | +-----------+ | |
.| | | | |
.| +-+-+-+ +--+-++
.+-+ | | |
. | L0 | | L1 |
. +-----+ +-----+Figure 2: Level Shortcut
Okay so, RIFT is not strictly stuck to Clos topologies 💅✨. The protocol only needs a sense of "compass rose directionality," which you can get either through configuration or by deriving the levels. So conceptually, shortcuts between levels could be included 🧭. Figure 2 shows an example of exactly that. In this example, suboptimal routing will happen when traffic goes from L0 to L1 via S0's default route and then back down through A0 or A1 😭. To avoid that, only default routes from A0 or A1 are used. All leaves would be required to install each other's routes 🌸.
Various technical and operational challenges may call for these kinds of modifications, but talking through those topics is outside the scope of this document 🙈.
4.3.2. Metro Networks 🌆
The demand for bandwidth keeps climbing, and it's being pushed mainly by environments close to content producers (server farms connecting via DC fabrics), but also by stuff near content consumers 📈. Consumers are often clustered in metro areas with their own network architectures, and those can benefit from simplified, regular Clos structures 💜. So, they can also benefit from RIFT 🦋.
4.3.3. Building Cabling 🏢
Commercial buildings are often cabled in topologies that are either Clos or their isomorphic equivalents 🧱. A Clos can grow pretty tall with lots of levels, and that's a real challenge for classical routing protocols. The exceptions are BGP [RFC4271] and Private Network-Network Interface (PNNI) [PNNI], which is mostly phased out by now 🥲. Those classical protocols don't support an arbitrary number of levels, but RIFT does that naturally 👑. Also, the forwarding tables in building-cabling network elements have limited sizes, so the minimum FIB size RIFT keeps under normal conditions is cost-effective in terms of hardware and operational costs 💰.
4.3.4. Internal Router Switching Fabrics 🔌
It's super common in high-speed communications switching and routing devices to use switch fabrics, which are interconnection networks inside the devices that connect input ports to output ports ⚡. A crossbar is one example of a switch fabric technique, even though it isn't feasible due to cost, head-of-line blocking, or size trade-offs 🙃. Normally, these fabrics are not self-healing or rely on 1:1 or 1+1 protection schemes. But it's conceivable to use RIFT to operate Clos fabrics that can deal effectively with interconnection or subsystem failures inside such a module 🛠️. RIFT is not IP specific, so any link addressing that connects internal device subnets is conceivable too 🌈.
4.3.5. CloudCO ☁️
The Cloud Central Office (CloudCO) is a brand new stage of the telecom Central Office 📡. It takes advantage of Software-Defined Networking (SDN) and Network Function Virtualization (NFV) together with general purpose hardware to optimize current networks 🔥. The figure below shows this architecture at a high level. It describes a single instance, or macro-node, of CloudCO that provides a number of value-added services (VASes), a Broadband Access Abstraction (BAA), and virtualized network services 🧚. An Access I/O module faces a CloudCO access node and the Customer Premises Equipment (CPE) behind it. A Network I/O module faces the core network. The two I/O modules are interconnected by a spine-and-leaf fabric [TR-384] 🫶.
+---------------------+ +----------------------+
| Spine | | Spine |
| Switch | | Switch |
+------+---+------+-+-+ +--+-+-+-+-----+-------+
| | | | | | | | | | | |
| | | | | +-------------------------------+ |
| | | | | | | | | | | |
| | | | +-------------------------+ | | |
| | | | | | | | | | | |
| | +----------------------+ | | | | | | | |
| | | | | | | | | | | |
| +---------------------------------+ | | | | | | |
| | | | | | | | | | | |
| | | +-----------------------------+ | | | | |
| | | | | | | | | | | |
| | | | | +--------------------+ | | | |
| | | | | | | | | | | |
+--+ +-+---+--+ +-+---+--+ +--+----+--+ +-+--+--+ +--+
|L | | Leaf | | Leaf | | Leaf | | Leaf | |L |
|S | | Switch | | Switch | | Switch | | Switch| |S |
++-+ +-+-+-+--+ +-+-+-+--+ +--+-+--+--+ ++-+--+-+ +-++
| | | | | | | | | | | | | |
| +-+-+-+--+ +-+-+-+--+ +--+-+--+--+ ++-+--+-+ |
| |Compute | |Compute | | Compute | |Compute| |
| |Node | |Node | | Node | |Node | |
| +--------+ +--------+ +----------+ +-------+ |
| || VAS5 || || vDHCP|| || vRouter|| ||VAS1 || |
| |--------| |--------| |----------| |-------| |
| |--------| |--------| |----------| |-------| |
| || VAS6 || || VAS3 || || v802.1x|| ||VAS2 || |
| |--------| |--------| |----------| |-------| |
| |--------| |--------| |----------| |-------| |
| || VAS7 || || VAS4 || || vIGMP || ||BAA || |
| |--------| |--------| |----------| |-------| |
| +--------+ +--------+ +----------+ +-------+ |
| |
++-----------+ +---------++
|Network I/O | |Access I/O|
+------------+ +----------+Figure 3: CloudCO Architecture Example
The Spine-Leaf architecture deployed inside CloudCO meets the network requirements of being adaptable, agile, scalable, and dynamic 🌟.
5. Operational Considerations 📋
RIFT gives organizations building and operating IP fabrics a bunch of features that simplify operation and deployment, while still getting many desirable properties of a dynamic routing protocol on that substrate 💖:
- RIFT only floods routing information to the devices that need it 🎯.
- RIFT allows for ZTP within the protocol. In its most extreme version, RIFT does not rely on any specific addressing and can operate using IPv6 Neighbor Discovery (ND) [RFC4861] only for IP fabric 🧁.
- RIFT has provisions to detect common IP fabric miscabling scenarios 🕵️♀️.
- RIFT automatically negotiates Bidirectional Forwarding Detection (BFD) per link. This allows for IP and micro-BFD [RFC7130] to replace Link Aggregation Groups (LAGs) that hide bandwidth imbalances in case of constituent failures 🔍. Further automatic link validation techniques similar to those in [RFC5357] could be supported as well ✅.
- RIFT inherently solves many problems associated with the use of classical routing topologies with dense meshes and high degrees of ECMP. It does this by including automatic bandwidth balancing, flood reduction, and automatic disaggregation on failures, while providing maximum aggregation of prefixes in default scenarios 🍓. ECMP in RIFT eliminates the need for more Loop-Free Alternate (LFA) procedures 🙌.
- RIFT reduces FIB size towards the bottom of the IP fabric, where most nodes reside 💅. This allows for cheaper hardware on the edges and the introduction of modern IP fabric architectures that encompass server multihoming and other mechanisms 💸.
- RIFT provides valley-free routing that is loop free 🔁. A valley-free path allows for reversal of direction at most once, from a packet heading northbound to southbound, while permitting traversal of horizontal links in the northbound phase. This allows for the use of any such valley-free path in bisectional fabric bandwidth between two destinations, irrespective of their metrics, which can be used to balance load on the fabric in different ways ⚖️. Valley-free routing eliminates the need for any specific micro-loop avoidance procedures for RIFT 🙅♀️.
- RIFT includes a key-value distribution mechanism that allows for future applications, such as automatic provisioning of basic overlay services or automatic key rollovers over whole fabrics 🗝️.
- RIFT is designed for minimum delay in case of prefix mobility on the fabric. In conjunction with [RFC8505], RIFT can differentiate anycast advertisements from mobility events and retain only the most recent advertisement in the latter case 🏃♀️💨.
- Many further operational and design points, collected over many years of routing protocol deployments, have been incorporated in RIFT, such as fast flooding rates, protection of information lifetimes, and operationally recognizable remote ends of links and node names 🥰.
5.1. South Reflection 🪞
South reflection is a mechanism where South Node TIEs are "reflected" back up north, so nodes in the same level without East-West links can "see" each other 👀.
For example, in Figure 4, Spine111\Spine112\Spine121\Spine122 reflects Node S-TIEs from ToF21 to ToF22 separately. Respectively, Spine111\Spine112\Spine121\Spine122 reflects Node S-TIEs from ToF22 to ToF21 separately, so ToF22 and ToF21 see each other's node information as level 2 nodes 💫.
In an equivalent fashion, as the result of the south reflection between Spine121-Leaf121-Spine122 and Spine121-Leaf122-Spine122, Spine121 and Spine 122 know each other at level 1 🎉.
5.2. Suboptimal Routing on Link Failures 🦋
okay so this one is about what happens when a link dies and the traffic takes the scenic route 🙈 check out Figure 4 👀
+--------+ +--------+
| ToF21 | | ToF22 | LEVEL 2
++--+-+-++ ++-+--+-++
| | | | | | | +
| | | | | | | linkTS8
+------------+ | +-+linkTS3+-+ | | | +-------------+
| | | | | | + |
| +---------------------------+ | linkTS7 |
| | | | + + + |
| | | +-------+linkTS4+------------+ |
| | | + + | | |
| | | +-------------+--+ | |
| | | | | linkTS6 | |
+-+----+-+ +-+----+-+ ++--------+ +-+----+-+
|Spine111| |Spine112| |Spine121 | |Spine122| LEVEL 1
+-+---+--+ +-+----+-+ +-+---+---+ +-+----+-+
| | | | | | | |
| +-------------+ | + ++XX+linkSL6+---+ +
| | | | linkSL5 | | linkSL8
| +-----------+ | | + +---+linkSL7+-+ | +
| | | | | | | |
+-+---+-+ +--+--+-+ +-+---+-+ +--+--+-+
|Leaf111| |Leaf112| |Leaf121| |Leaf122| LEVEL 0
+-+-----+ +-+-----+ +-----+-+ +-+-----+
+ + + +
Prefix111 Prefix112 Prefix121 Prefix122Figure 4: Suboptimal Routing Upon Link Failure Use Case
okay so, thanks to the south reflection, Spine121 and Spine 122 totally know each other through Leaf121 or Leaf 122 at level 1 🫶
Without disaggregation mechanisms, the packet from leaf121 to prefix122 will probably go up through linkSL5 to linkTS3 when linkSL6 fails. Then it goes down through linkTS4 to linkSL8 to Leaf122, or it goes up through linkSL5 to linkTS6, then down through linkTS8 and linkSL8 to Leaf122, based on the pure default route. This is the case of suboptimal routing (the spec calls it "bow tying" 🎀), and honestly it's giving detour energy 😭
With disaggregation mechanisms, Spine122 will detect the failure according to the reflected node S-TIE from Spine121 when linkSL6 fails. Based on the disaggregation algorithm provided by RIFT, Spine122 will explicitly advertise prefix122 in a Disaggregated Prefix S-TIE, PrefixTIEElement(prefix122, cost 1). The packet from leaf121 to prefix122 will only be sent to linkSL7 following a longest-prefix match to prefix 122 directly, then it goes down through linkSL8 to Leaf122. main character routing, no detours 💅✨
5.3. Black-Holing on Link Failures 🕳️
this section is the scary one 💀 it covers a double link failure, and that's when black-holing can happen
+--------+ +--------+
| ToF 21 | | ToF 22 | LEVEL 2
++-+--+-++ ++-+--+-++
| | | | | | | +
| | | | | | | linkTS8
+--------------+ | +-+linkTS3+X+ | | | +--------------+
linkTS1 | | | | | + |
+ +-----------------------------+ | linkTS7 |
| | + | + + + |
| | linkTS2 +-------+linkTS4+X+----------+ |
| + + + + | | |
| linkTS5 +-+ +------------+--+ | |
| + | | | linkTS6 | |
+-+----+-+ +-+----+-+ ++-------+ +-+-----++
|Spine111| |Spine112| |Spine121| |Spine122| LEVEL 1
+-+---+--+ ++----+--+ +-+---+--+ +-+----+-+
| | | | | | | |
+ +---------------+ | + +---+linkSL6+---+ +
linkSL1 | | | linkSL5 | | linkSL8
+ +--+linkSL3+--+ | | + +---+linkSL7+-+ | +
| | | | | | | |
+-+---+-+ +--+--+-+ +-+---+-+ +--+--+-+
|Leaf111| |Leaf112| |Leaf121| |Leaf122| LEVEL 0
+-+-----+ +-+-----+ +-----+-+ +-----+-+
+ + + +
Prefix111 Prefix112 Prefix121 Prefix122Figure 5: Black-Holing Upon Link Failure Use Case
so without disaggregation mechanisms, the packet from leaf111 to prefix122 would suffer 50% black-holing based on the pure default route when linkTS3 and linkTS4 both fail 😱 the packet is supposed to go up through linkSL1 to linkTS1, and then if it goes down through linkTS3 or linkTS4, it gets dropped. the packet is also supposed to go up through linkSL3 to linkTS2, and then if it goes down through linkTS3 or linkTS4, it gets dropped as well. this is the case of black-holing, and it's a whole vibe-killer 🥲
With disaggregation mechanisms, ToF22 will detect the failure according to the reflected node S-TIE of ToF21 from Spine111\Spine112 when linkTS3 and linkTS4 both fail. Based on the disaggregation algorithm provided by RIFT, ToF22 will explicitly originate an S-TIE with prefix 121 and prefix 122, which is flooded to spines 111, 112, 121, and 122. ToF22 really said "I got this" 🫡
The packet from leaf111 to prefix122 will not be routed to linkTS1 or linkTS2. The packet from leaf111 to prefix122 will only be routed to linkTS5 or linkTS7 following a longest-prefix match to prefix122. no more dropped packets, bestie 🎉
5.4. Zero Touch Provisioning (ZTP) 🪄
RIFT is designed to need a very minimal configuration, which keeps things simple and avoids human errors 🙌 based on that tiny bit of info, ZTP auto-configures the key operational parameters of all the RIFT nodes. That includes the System ID of the node, which must be unique in the RIFT network, and the level of the node in the Fat Tree. The level decides which peers are northward "parents" and which are southward "children" 🌳
ZTP is always on, but its decisions can be overridden when a network administrator wants to impose their own configuration. In that case, it is the responsibility of the administrator to ensure that the configured parameters are correct, i.e., ensure that the System ID of each node is unique and that the administratively set levels truly reflect the relative position of the nodes in the fabric. It is recommended to let ZTP configure the network. When ZTP does not configure the network, it is recommended to configure the level of all the nodes, to avoid an undesirable interaction between ZTP and the manual configuration. so if you go manual, you do you, but be careful 💅
ZTP requires that the administrator points out the ToF nodes to set the baseline from which the fabric topology is derived. The ToF nodes are configured with the TOP_OF_FABRIC flag. They're the initial "seeds" that other ZTP nodes need in order to derive their level in the topology 🌱 ZTP computes the level of each node based on the Highest Available Level (HAL) of the potential parent closest to that baseline, which represents the superspine. in a way, RIFT is giving distance-vector protocol energy: it computes a set of feasible successors towards the superspine and autoconfigures the rest of the topology 🧠✨
The autoconfiguration mechanism computes a global maximum of levels by diffusion. The derivation of each node's level then happens based on LIEs received from its neighbors, and each node (with possible exceptions of configured leaves) tries to attach at the highest possible point in the fabric. This guarantees that even if the diffusion front reaches a node from "below" faster than from "above", it will greedily abandon already negotiated levels derived from nodes topologically below it and properly peer with nodes above. it's the glow-up arc, no cap 🌈
The achieved equilibrium can be massively disturbed by all nodes with the highest level either leaving or entering the domain (with some finer distinctions not explained further). It is therefore recommended that each node is multihomed towards nodes with respective HAL offerings. Fortunately, this is the natural state of things for the topology variants considered in RIFT, so we're lowkey fine 🙏
A RIFT node may also be configured to confine itself to the leaf role with the LEAF_ONLY flag. A leaf node can also be configured to support leaf-2-leaf procedures with the LEAF_2_LEAF flag. In both cases, the node cannot be TOP_OF_FABRIC, and its level cannot be configured. RIFT will fully determine the node's level after it is attached to the topology, and ensure that the node is at the "bottom of the hierarchy" (southernmost) 🦋
5.5. Miscabling 🧚
5.5.1. Miscabling Examples ✨
Okay bestie, let's look at the miscabling examples 🔍 Figure 6 shows a single-plane miscabling example. Honestly it's a perfect Fat Tree fabric 💅 except for one sneaky link, link-M, which connects Leaf112 to ToF22. 🙈
+----------------+ +-----------------+
| ToF21 | +------+ ToF22 | LEVEL 2
+-------+----+---+ | +----+---+--------+
| | | | | | | | |
| | | +----------------------------+ |
| +---------------------------+ | | | |
| | | | | | | | |
| | | | +-----------------------+ | |
| | +------------------------+ | | |
| | | | | | | | |
+-+---+--+ +-+---+--+ | +--+---+-+ +--+---+-+
|Spine111| |Spine112| | |Spine121| |Spine122| LEVEL 1
+-+---+--+ ++----+--+ | +--+---+-+ +-+----+-+
| | | | | | | | |
| +---------+ | link-M | +---------+ |
| | | | | | | | |
| +-------+ | | | | +-------+ | |
| | | | | | | | |
+-+---+-+ +--+--+-+ | +-+---+-+ +--+--+-+
|Leaf111| |Leaf112+-----+ |Leaf121| |Leaf122| LEVEL 0
+-------+ +-------+ +-------+ +-------+
Figure 6: A Single-Plane Miscabling ExampleThe RIFT control protocol is seriously impressive 🤩 It can discover the physical links automatically and detect cabling that breaks Fat Tree topology constraints. When it spots a miscabling attempt, it reacts: it prevents adjacencies between nodes from forming and, at a minimum, stops traffic from being forwarded over those miscabled links. 🚫 In this scenario, Leaf112 uses link-M to derive its level (unless it is a leaf), and it can report its links to Spine111 and Spine112 as miscabled, unless the implementation allows horizontal links.
Figure 7 is the multi-plane miscabling example 🦋 Leaf112 and Spine121 belong to two different PoDs, so the adjacency between them cannot be formed. Link-W would be detected and prevented. Iconic behaviour ✨
+-------+ +-------+ +-------+ +-------+
|ToF A1| |ToF A2| |ToF B1| |ToF B2| LEVEL 2
+-------+ +-------+ +-------+ +-------+
| | | | | | | |
| | | +-----------------+ | | |
| +--------------------------+ | | | |
| +------+ | | | +------+ |
| | +-----------------+ | | | | |
| | | +--------------------------+ | |
| A | | B | | A | | B |
+-----+--+ +-+---+--+ +--+---+-+ +--+-----+
|Spine111| |Spine112| +---+Spine121| |Spine122| LEVEL 1
+-+---+--+ ++----+--+ | +--+---+-+ +-+----+-+
| | | | | | | | |
| +---------+ | | | +---------+ |
| | | | link-W | | | |
| +-------+ | | | | +-------+ | |
| | | | | | | | |
+-+---+-+ +--+--+-+ | +-+---+-+ +--+--+-+
|Leaf111| |Leaf112+------+ |Leaf121| |Leaf122| LEVEL 0
+-------+ +-------+ +-------+ +-------+
+--------PoD#1----------+ +---------PoD#2---------+
Figure 7: A Multiple Plane Miscabling ExampleRIFT also has an optional level determination procedure in its ZTP mode 🧚 Nodes in the fabric without their level configured figure it out automatically. 🤔 However, this can have some counter-intuitive consequences. One extreme failure scenario is shown in Figure 8: if all northbound links of Spine11 fail at the same time, Spine11 negotiates a lower level than Leaf11 and Leaf12. Wild, right? 😭
+-------+ +-------+ +-------+ +-------+
|ToF A1| |ToF A2| |ToF A1| |ToF A2|
+-------+ +-------+ +-------+ +-------+
| | | | | |
| +-------+ | | |
+ + | | ====> | |
X X +------+ | +------+ |
+ + | | | |
+----+--+ +-+-----+ +-+-----+
|Spine11| |Spine12| |Spine12|
+-+---+-+ ++----+-+ ++----+-+
| | | | | |
| +---------+ | | |
| +-------+ | | +-------+ |
| | | | | |
+-+---+-+ +--+--+-+ +-----+-+ +-----+-+
|Leaf111| |Leaf112| |Leaf111| |Leaf112|
+-------+ +-------+ +-+-----+ +-+-----+
| |
| +--------+
| |
+-+---+-+
|Spine11|
+-------+
Figure 8: Fallen SpineTo prevent that scenario, where leaves are expected to act as switches 🙅♀️, the LEAF_ONLY flag can be set for Leaf111 and Leaf112. Since level -1 is invalid, Spine11 would not derive a valid level from the topology in Figure 8. It gets isolated from the whole fabric, and it's up to the leaves to declare the links towards that spine as miscabled. Main character energy for the leaves 👑
5.5.2. Miscabling Considerations 🎀
Some operators want to use ZTP and add extra cabling constraints that go beyond the topology violations we just covered 💖 Enforcing cabling down to specific level, node, and port combinations can make life way easier for onsite staff. It simplifies troubleshooting, and it makes replacing optical transceivers and/or cabling simpler too, because the physical layout stays consistent across the fabric. This is especially true for densely connected fabrics, where it's really hard to physically manipulate those components. It's also easy to imagine other models, like one where the strict port requirement is relaxed. 🌈
Figure 9 shows an example where the first port on Leaf1 must connect to the first port on Spine1, the second port on Leaf1 must connect to the first port on Spine2, and so on. Now picture (Leaf1, Port1) and (Leaf1, Port2) being reversed. By default, RIFT would not consider that miscabled, but an operator might want it to be 🧐
+--------+ +--------+ +--------+ +--------+
| Spine1 | | Spine2 | | Spine3 | | Spine4 |
+-1------+ +-1------+ +-1------+ +-1------+
+ + + +
| +----------+ | |
| | | |
| | +---------------------+ |
| | | |
| | | +--------------------------------+
| | | |
| | | |
| | | |
| | | |
+ + + +
+-1--2--3--4--+
| Leaf1 | ......
+-------------+
Figure 9: Additional Cabling Constraint ExampleRIFT lets implementations provide programmable plug-ins that can adjust ZTP operation or capture information during computation. Defining that is outside the scope of this document, but such a mechanism could be used to extend the miscabling functionality ✨
Other protocols would need extra operational overhead to get this kind of behaviour 😩 Take a fabric using unnumbered OSPF links: a miscabled link will still very likely form an adjacency. Every attempt to move cables to the correct port may mean more troubleshooting, because other links become miscabled in the process. Without automation that explicitly tells the operator which ports need to be moved where, the process gets manually intensive and error-prone very quickly. 💀 If the problem goes unnoticed, it results in suboptimal performance in the fabric.
5.6. Multicast and Broadcast Implementations 📡
RIFT supports both multicast and broadcast implementations 🫶 A multicast implementation is preferred, but there might be cases where a broadcast implementation is optimal or even required. For example, operating systems on IoT devices and embedded devices may not have the required multicast support. Containers are another example: they do support multicast in some cases, but they tend to be very CPU-inefficient and difficult to tune. 🍓
5.7. Positive vs. Negative Disaggregation
okay so disaggregation is the thing where RIFT [RFC9692] sends a more specific route southwards as an exception to the aggregated fabric-default north 💅✨ it's giving "hey, I know a better path for this one" energy 🦋. It's useful when a prefix inside the aggregation is reachable through some of the parents at the same level but not the others 🌸. It is mandatory when the level is the ToF, because a ToF node that can't reach a prefix turns into a black hole for that prefix 💀. The hard part is knowing which prefixes are reachable by whom 🙈.
In the general case, RIFT [RFC9692] solves that by interconnecting the ToF nodes so they can swap the full list of prefixes that exist in the fabric and figure out when one of them lacks reachability to some prefixes 🔥. This needs extra ports at the ToF, typically two ports per ToF node to form a ToF-spanning ring 💍. RIFT [RFC9692] also defines the southbound reflection procedure, which lets a parent explore the direct connectivity of its peers (their own parents and children) 👀. Based on the advertisements it gets from the shared parents and children, it may let the parent infer which prefixes its peers can reach 🧚.
When a parent lacks reachability to a prefix, it may disaggregate that prefix negatively, meaning it advertises that this parent can be used to reach any prefix in the aggregation except that one 🙃. Negative Disaggregation signaling is simple and works transitively from ToF to Top-of-Pod (ToP) and then from ToP to Leaf 💜. The catch is that it's hard for a parent to figure out which prefix it needs to disaggregate, because it doesn't know what it doesn't know 😭. So using a spanning ring at the ToF is required to operate Negative Disaggregation 🫶. Also, though it's only an implementation problem, programming the FIB is complex compared to normal routes and may incur recursions 🤯.
The more classical alternative is for the parents that can reach a prefix (when their peers at the same level can't) to advertise a more specific route to that prefix 🎀. This leans on the normal longest prefix match in the FIB and doesn't need any special implementation 💖. Compared to Negative Disaggregation, Positive Disaggregation is difficult and inefficient to operate transitively 😬.
Transitivity isn't needed by a grandchild if all its parents received the Positive Disaggregation, meaning they shall all avoid the black hole 🛡️. When that's the case, they collectively build a ceiling that protects the grandchild 👑. Until then, a parent that received the Positive Disaggregation may think some peers are lacking reachability and re-advertise too early, or it may defer and keep a black hole situation going longer than necessary 🙈.
In a non-partitioned fabric, all the ToF nodes see one another through the reflection and can figure out whether one is missing a child 👯♀️. In that case, it's possible to compute the prefixes the peer can't reach and disaggregate positively without a ToF-spanning ring 🌈. The ToF nodes can also ascertain that each ToP node is connected to at least one ToF node that can still reach the prefix, which means the transitive operation isn't required 🧠✨.
The bottom line: in a fabric that is partitioned (e.g., using multiple planes) and/or where the ToP nodes are not guaranteed to always form a ceiling for their children, it is mandatory to use Negative Disaggregation 🚨. On the other hand, in a highly symmetrical and fully connected fabric (e.g., a canonical Clos Network), Positive Disaggregation methods save the complexity and cost associated with the ToF-spanning ring 💸✨.
Heads up: with Positive Disaggregation, the first ToF nodes that announce a more-specific route attract all the traffic for that route and may suffer a transient incast 🌪️. A ToP node that defers injecting the longer prefix into the FIB (to receive more advertisements and spread the packets better) also keeps sending a portion of the traffic to the black hole in the meantime 😩. With Negative Disaggregation, the last ToF nodes that inject the route may also hit an incast issue; this would happen if a prefix that becomes totally unreachable gets disaggregated 🥺.
5.8. Mobile Edge and Anycast
When a physical or virtual node changes its point of attachment in the fabric from a previous-leaf to a next-leaf, new routes must be installed that supersede the old ones 🏃♀️💨. Since the flooding flows northwards, the nodes (if any) between the previous-leaf and the common parent aren't immediately aware that the path via the previous-leaf is obsolete, so a stale route may hang around for a while 👻. The common parent needs to select the freshest route advertisement so it can install the correct route via the next-leaf 🎯. That means the fabric has to determine the sequence of the mobile node's movements 🧩.
On the one hand, a classical sequence counter gives a total order for a while, but it will eventually wrap 🔄. On the other hand, a timestamp gives a permanent order, but it may miss a movement that happens too quickly compared to the granularity of the timing information ⏱️. It's not envisioned that an average fabric supports the Precision Time Protocol [IEEEstd1588] in the short term, and the precision available with the Network Time Protocol [RFC5905] (on the order of 100 to 200 ms) may not necessarily be enough to cover things like the fast mobility of a Virtual Machine (VM) 🖥️💨.
Section 6.8.4 ("Mobility") of [RFC9692] specifies a hybrid method that combines a sequence counter from the mobile node with a timestamp from the network, taken at the leaf when the route is injected 🤝. If the timestamps of the concurrent advertisements are comparable (i.e., more distant than the precision of the timing protocol), then the timestamp alone is used to determine the relative freshness of the routes 🕰️. Otherwise, the sequence counter from the mobile node is used if it's available 🔢. One caveat: the sequence counter must not wrap within the precision of the timing protocol ⚠️. Another: the mobile node may not even provide a sequence counter, in which case the mobility itself must be slower than the precision of the timing 🐢.
Mobility must not be confused with anycast 🙅♀️. In both cases, the same address is injected in RIFT at different leaves. With mobility, only the freshest route must be conserved, since the mobile node changes its point of attachment from one leaf to the next 🚶♀️. With anycast, the node may either be multihomed (attached to multiple leaves in parallel) or reachable beyond the fabric via multiple routes that are redistributed to different leaves 🌐. Either way, the multiple routes are equally valid and should be conserved in the case of anycast 💯. Without further information from the redistributed routing protocol, it's impossible to sort out a movement from a redistribution that happens asynchronously on different leaves 😵💫. RIFT [RFC9692] expects anycast addresses to be advertised within the timing precision, which is typically the case with low-precision timing and a multihomed node 🙌. Beyond that time interval, RIFT interprets the lag as mobility and only the freshest route is retained 🏁.
When using IPv6 [RFC8200], RIFT suggests leveraging 6LoWPAN ND [RFC8505] as the IPv6 ND interaction between the mobile node and the leaf 📡. This gives you not only a sequence counter but also a lifetime and a security token that may be used to protect the ownership of an address [RFC8928] 🔐✨. When using 6LoWPAN ND [RFC8505], the parallel registration of an anycast address to multiple leaves is done with the same sequence counter, whereas the sequence counter is incremented when the point of attachment changes 🔁. This way it's possible to tell a mobile node apart from a multihomed node, even when the mobility happens within the timing precision 🕵️♀️💅. It's also possible for a mobile node to be multihomed as well, e.g., to change only one of its points of attachment 🎀.
5.9. IPv4 over IPv6
RIFT allows advertising IPv4 prefixes over an IPv6 RIFT network 🌏. An IPv6 Address Family (AF) is configured via the usual ND mechanisms, and then V4 can use V6 next-hops analogous to [RFC8950] 🧩. It is expected that the whole fabric supports the same type of forwarding of AFs on all the links 🔗. RIFT provides an indication of whether a node is capable of V4-forwarding, and implementations are possible where different routing tables are computed per AF as long as the computation remains loop-free 🔄💖.
+-----+ +-----+
+---+---+ | ToF | | ToF |
^ +--+--+ +-----+
| | | | |
| | +-------------+ |
| | +--------+ | |
+ | | | |
V6 +-----+ +-+---+
Forwarding |Spine| |Spine|
+ +--+--+ +-----+
| | | | |
| | +-------------+ |
| | +--------+ | |
| | | | |
v +-----+ +-+---+
+---+---+ |Leaf | | Leaf|
+--+--+ +--+--+
| |
IPv4 prefixes| |IPv4 prefixes
| |
+---+----+ +---+----+
| V4 | | V4 |
| subnet | | subnet |
+--------+ +--------+
Figure 10: IPv4 over IPv65.10. In-Band Reachability of Nodes ✨
okay so RIFT doesn't require the nodes in the fabric to have reachable addresses 🙈 but there are still totally real operational reasons you'd want to reach the internal nodes. Figure 11 shows the network management station (NMS) plugged into Leaf1 💻
+-------+ +-------+
| ToF1 | | ToF2 |
++---- ++ ++-----++
| | | |
| +----------+ |
| +--------+ | |
| | | |
++-----++ +--+---++
|Spine1 | |Spine2 |
++-----++ ++-----++
| | | |
| +----------+ |
| +--------+ | |
| | | |
++-----++ +--+---++
| Leaf1 | | Leaf2 |
+---+---+ +-------+
|
|NMS
Figure 11: In-Band Reachability of Nodesso if the NMS wants to reach Leaf2, it just works 💅 because Leaf2's loopback address gets flooded in its Prefix North TIE.
if the NMS wants Spine2, that also works 🦋 because a spine node always advertises its loopback address in the Prefix North TIE. The NMS can get to Spine2 via Leaf1-Spine2, or the long way around through Leaf1-Spine1-ToF1/ToF2-Spine2. main character energy 👑
now ToF2 is the tricky one 😭 ToF2's loopback address has to be injected into its Prefix South TIE. That TIE has to be seen by every node at the level below (the spine nodes in Figure 11), because those spines need to form a ceiling for all the traffic coming up from below (south). If they don't, traffic from the NMS might follow the default route to the wrong ToF node, like ToF1. yikes 💀
and if the link between ToF2 and the spine nodes fails, ToF2's loopback address has to be disaggregated recursively all the way down to the leaves 🌈 even then, in a partitioned ToF, a ToF node is still only reachable within its own plane. not ideal, bestie 🥺
there's also a possible alternative to recursive disaggregation 🧚 you can use a ring that connects the ToF nodes so they can pass packets between each other for their loopback addresses only. the vibe is that this is mostly control traffic, so it shouldn't mess with the load-balancing properties of the fabric 🫶
5.11. Dual-Homing Servers 🎀
okay so each RIFT node may run in ZTP mode 🧚 it starts with no configuration (unless it's a ToF node at the top of the topology, or it has to operate in the topology as a leaf and/or support leaf-2-leaf procedures), and then it fully configures itself once it's attached to the topology. literally so easy 🍓
+---+ +---+ +---+
|ToF| |ToF| |ToF| ToF
+---+ +---+ +---+
| | | | | |
| +----------------+ | |
| +----------------+ |
| | | | | |
+----------+--+ +--+----------+
| ToR1 | | ToR2 | Spine
+--+------+---+ +--+-------+--+
+---+ | | | | | | +---+
| +-----------------+ | | |
| | | +-------------+ | |
| | | | | +-----------------+ |
| | | | +--------------+ | | |
| | | | | | | |
+---+ +---+ +---+ +---+
| | | | | | | |
+---+ +---+ ............. +---+ +---+
SV(1) SV(2) SV(n-1) SV(n) Leaf
Figure 12: Dual-Homing Serverssometimes people want to disaggregate from the ToR nodes down to the servers right from startup 💖 so the servers have multiple routes in the FIB from the beginning, other than the default route, to avoid breakages at the rack level. full disaggregation of the fabric can be set up with configuration that RIFT supports ⭐
5.12. Fabric with a Controller 🦋
there are lots of ways to deploy the controller ✨ one option is attaching it to the RIFT domain from the ToF, and another is attaching it from the leaf.
+------------+
| Controller |
++----------++
| |
| |
+----++ ++----+
------- | ToF | | ToF |
| +--+--+ +-----+
| | | | |
| | +-------------+ |
| | +--------+ | |
| | | | |
+-----+ +-+---+
RIFT domain |Spine| |Spine|
+--+--+ +-----+
| | | | |
| | +-------------+ |
| | +--------+ | |
| | | | |
| +-----+ +-+---+
------- |Leaf | | Leaf|
+-----+ +-----+
Figure 13: Fabric with a Controller5.12.1. Controller Attached to ToFs 💅
if a controller is attaching to the RIFT domain from a ToF, it usually uses dual-homing connections 🫶 the controller's loopback prefix should be advertised down by the ToF and spine nodes to the leaves. and if the controller loses its link to the ToF, make sure the ToF withdraws the controller's prefix 🙏
5.12.2. Controller Attached to Leaf 🌸
if the controller attaches from a leaf to the fabric, no special provisions are needed. easy peasy 🍓
5.13. Internet Connectivity Within Underlay 🌍
if global addressing is running without an overlay, an external default route needs to be advertised through the RIFT fabric so you can reach the internet 🌐 and for forwarding across the entire RIFT fabric, an internal fabric prefix needs to be advertised in the Prefix South TIE by the ToF and spine nodes.
5.13.1. Internet Default on the Leaf 🍃
if the internet gateway is a leaf, that leaf needs to advertise a default route in its Prefix North TIE 💜
5.13.2. Internet Default on the ToFs 🌟
if the internet gateway is a ToF, the ToF and spine nodes need to advertise a default route in the Prefix South TIE ✨
5.14. Subnet Mismatch and Address Families 🧩
+--------+ +--------+
| | LIE LIE | |
| A | +----> <----+ | B |
| +---------------------+ |
+--------+ +--------+
X/24 Y/24
Figure 14: Subnet MismatchLIEs get exchanged over all links running RIFT to do Link (Neighbor) Discovery 🔍 a node MUST NOT originate LIEs on an AF if it doesn't process received LIEs on that family 🚫 LIEs on the same link count as part of the same negotiation, no matter which AF they arrive on. and an implementation must be ready to accept TIEs on all addresses it used as the source of LIE frames 💖
so as shown in Figure 14, an adjacency between nodes A and B can form without any further checks 😬 but forwarding between A and B may fail, because subnet X doesn't match subnet Y. ugh, the worst 💀
to prevent this, a RIFT implementation should check for subnet mismatch in a way that's similar to how IS-IS does it 🧠 this can lead to scenarios where an adjacency, even though LIEs got exchanged in both AFs, ends up being an adjacency in a single AF only. this is especially something to think about in scenarios related to Section 5.9 👀
5.15. Anycast Considerations 📡
+ traffic
|
v
+------+------+
| ToF |
+---+-----+---+
| | | |
+------------+ | | +------------+
| | | |
+---+---+ +-------+ +-------+ +---+---+
| | | | | | | |
|Spine11| |Spine12| |Spine21| |Spine22| LEVEL 1
+-+---+-+ ++----+-+ +-+---+-+ ++----+-+
| | | | | | | |
| +---------+ | | +---------+ |
| +-------+ | | | +-------+ | |
| | | | | | | |
+-+---+-+ +--+--+-+ +-+---+-+ +--+--+-+
| | | | | | | |
|Leaf111| |Leaf112| |Leaf121| |Leaf122| LEVEL 0
+-+-----+ ++------+ +-----+-+ +-----+-+
+ + + ^ +
PrefixA PrefixB PrefixA | PrefixC
|
+ traffic
Figure 15: Anycastso if traffic comes from the ToF to Leaf111 or Leaf121, which both have the anycast prefix PrefixA, RIFT handles that case really well 🙌 but if the traffic comes from Leaf122, it arrives at Spine21 or Spine22 at LEVEL 1. on top of that, Spine21 or Spine22 doesn't know about the other PrefixA attached to Leaf111, so it'll always get to Leaf121 and never Leaf111 😭 if the intention is for the traffic to be offloaded to Leaf111, then use the policy-guided prefixes defined in RIFT [RFC9692] 🎀
5.16. IoT Applicability 🧸
okay so RIFT basically inherits its whole "default route goes upwards (northwards)" energy from RPL [RFC6550] 🌲. It also gets the ability to inject external host routes at the Leaf level using Wireless ND (WiND) [RFC8505] [RFC8928] between a RIFT-agnostic host and a RIFT router. Both RPL and RIFT are built for large scale, and WiND makes device mobility at the edge work the same way in both 🦋.
The big difference is that RPL has a single root, but RIFT has many ToF nodes. ngl that's huge for leaf-2-leaf ECMP paths 🔥, but it also brings extra complexity because you need to disaggregate. Also, RIFT does link-state flooding northwards and is not designed for low-power operation 🔋.
Still, nothing stops the IP devices connected at the Leaf from being IoT devices, which usually expose their address using WiND. This is an upgrade from 6LoWPAN ND [RFC6775] 💖.
A network that serves high speed / high power IoT devices should typically provide deterministic capabilities for stuff like high speed control loops or movement detection. The Fat Tree is super reliable and, in normal conditions, gives you an equivalent multipath operation 🌈. However, the ECMP doesn't provide hard guarantees for either delivery or latency 😬. As long as the fabric is non-blocking, the result is the same, but there can be load unbalances that lead to incast and possibly congestion loss, which will prevent delivery within bounded latency 💀.
This could be alleviated with Packet Replication, Elimination, and Ordering Functions (PREOF) [RFC8655] leaf-2-leaf, but PREOF is hard to provide at the scale of all flows, and the replication may increase the probability of the overload it's trying to solve 🙈.
Note that the load balancing is not RIFT's problem, but it is key to serve IoT adequately 🎀.
5.17. Key Management 🔐
As outlined in Section 9 ("Security Considerations") of [RFC9692], either a private shared key or a public/private key pair is used to authenticate the adjacency. Both the key distribution and key synchronization methods are out of scope for this document. Both nodes in the adjacency must share the same keys, key type, and algorithm for a given key ID. Mismatched keys will not interoperate, since their security envelopes will be unverifiable 🙅♀️.
Key rollover while the adjacency is active may be supported. The specific mechanism is well documented in [RFC6518] 📖. As outlined in 9.9 ("Host Implementations") of [RFC9692], hosts as well as VMs acting as RIFT devices are possible 🖥️. Key Management Protocols (KMPs), such as Key Value (KV) for key rollover in the fabric, use a symmetric key that can be changed easily when it's compromised. In that case, the symmetric key of a host is more likely to be compromised than an in-fabric networking node 👀.
5.18. TTL/Hop Limit of 1 vs. 255 on LIEs/TIEs ⏲️
The use of a packet's Time to Live (TTL) (IPv4) or Hop Limit (IPv6) to verify whether the packet was originated by an adjacent node on a connected link has been used in RIFT. RIFT explicitly requires the use of a TTL/HL value of 1 or 255 when sending/receiving LIEs and TIEs, so that implementers get a choice between the two 🍓.
TTL=1 or HL=1 protects against the information disseminating more than 1 hop in the fabric, and should be the default unless configured otherwise. TTL=255 or HL=255 can lead RIFT TIE packet propagation to more than one hop (the multicast address is already in local subnetwork range) in case of implementation problems. It does protect against a remote attack as well, though, and the receiving remote router will ignore such TIE packet unless the remote router is exactly 254 hops away and accepts only TTL=1 or HL=1 🛡️. [RFC5082] defines a Generalized TTL Security Mechanism (GTSM). The GTSM is applicable to LIE/TIE implementations that use a TTL or HL of 255. It provides a defense from infrastructure attacks based on forged protocol packets from outside the fabric 🏰.
6. Security Considerations 🚨
This document presents applicability of RIFT, so it does not introduce any security considerations of its own. BUT there are a number of security concerns in [RFC9692], and they are seriously not optional. Go read them, bestie 💀.
7. IANA Considerations 📋
This document has no IANA actions.
8. References 📚
8.1. Normative References
[ISO10589-Second-Edition]
ISO/IEC, "Information technology - Telecommunications and
information exchange between systems - Intermediate System
to Intermediate System intra-domain routeing information
exchange protocol for use in conjunction with the protocol
for providing the connectionless-mode network service (ISO
8473)", ISO/IEC 10589:2002, November 2002,
<https://www.iso.org/standard/30932.html>.
[RFC2328] Moy, J., "OSPF Version 2", STD 54, RFC 2328,
DOI 10.17487/RFC2328, April 1998,
<https://www.rfc-editor.org/info/rfc2328>.
[RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
"Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
DOI 10.17487/RFC4861, September 2007,
<https://www.rfc-editor.org/info/rfc4861>.
[RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C.
Pignataro, "The Generalized TTL Security Mechanism
(GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007,
<https://www.rfc-editor.org/info/rfc5082>.
[RFC5340] Coltun, R., Ferguson, D., Moy, J., and A. Lindem, "OSPF
for IPv6", RFC 5340, DOI 10.17487/RFC5340, July 2008,
<https://www.rfc-editor.org/info/rfc5340>.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
RFC 5357, DOI 10.17487/RFC5357, October 2008,
<https://www.rfc-editor.org/info/rfc5357>.
[RFC6518] Lebovitz, G. and M. Bhatia, "Keying and Authentication for
Routing Protocols (KARP) Design Guidelines", RFC 6518,
DOI 10.17487/RFC6518, February 2012,
<https://www.rfc-editor.org/info/rfc6518>.
[RFC6550] Winter, T., Ed., Thubert, P., Ed., Brandt, A., Hui, J.,
Kelsey, R., Levis, P., Pister, K., Struik, R., Vasseur,
JP., and R. Alexander, "RPL: IPv6 Routing Protocol for
Low-Power and Lossy Networks", RFC 6550,
DOI 10.17487/RFC6550, March 2012,
<https://www.rfc-editor.org/info/rfc6550>.
[RFC6775] Shelby, Z., Ed., Chakrabarti, S., Nordmark, E., and C.
Bormann, "Neighbor Discovery Optimization for IPv6 over
Low-Power Wireless Personal Area Networks (6LoWPANs)",
RFC 6775, DOI 10.17487/RFC6775, November 2012,
<https://www.rfc-editor.org/info/rfc6775>.
[RFC7130] Bhatia, M., Ed., Chen, M., Ed., Boutros, S., Ed.,
Binderberger, M., Ed., and J. Haas, Ed., "Bidirectional
Forwarding Detection (BFD) on Link Aggregation Group (LAG)
Interfaces", RFC 7130, DOI 10.17487/RFC7130, February
2014, <https://www.rfc-editor.org/info/rfc7130>.
[RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas,
"Deterministic Networking Architecture", RFC 8655,
DOI 10.17487/RFC8655, October 2019,
<https://www.rfc-editor.org/info/rfc8655>.
[RFC8950] Litkowski, S., Agrawal, S., Ananthamurthy, K., and K.
Patel, "Advertising IPv4 Network Layer Reachability
Information (NLRI) with an IPv6 Next Hop", RFC 8950,
DOI 10.17487/RFC8950, November 2020,
<https://www.rfc-editor.org/info/rfc8950>.
[RFC9692] Przygienda, T., Ed., Head, J., Ed., Sharma, A., Thubert,
P., Rijsman, B., and D. Afanasiev, "RIFT: Routing in Fat
Trees", RFC 9692, DOI 10.17487/RFC9692, April 2025,
<https://www.rfc-editor.org/info/rfc9692>.
[TR-384] Broadband Forum Technical Report, "TR-384: Cloud Central
Office Reference Architectural Framework", TR-384, Issue
1, January 2018,
<https://www.broadband-forum.org/pdfs/tr-384-1-0-0.pdf>.8.2. Informative References
[CLOS] Yuan, X., "On Nonblocking Folded-Clos Networks in Computer
Communication Environments", 2011 IEEE International
Parallel & Distributed Processing Symposium,
DOI 10.1109/IPDPS.2011.27, May 2011,
<https://ieeexplore.ieee.org/document/6012836>.
[FATTREE] Leiserson, C. E., "Fat-Trees: Universal Networks for
Hardware-Efficient Supercomputing", IEEE Transactions on
Computers, vol. C-34, no. 10, pp. 892-901,
DOI 10.1109/TC.1985.6312192, October 1985,
<https://ieeexplore.ieee.org/document/6312192>.
[IEEEstd1588]
IEEE, "IEEE Standard for a Precision Clock Synchronization
Protocol for Networked Measurement and Control Systems",
IEEE Std 1588-2019, DOI 10.1109/IEEESTD.2020.9120376, June
2020, <https://ieeexplore.ieee.org/document/9120376>.
[PNNI] The ATM Forum Technical Committee, "Private Network-
Network Interface - Specification Version 1.1 - (PNNI
1.1)", af-pnni-0055.001, April 2002,
<https://www.broadband-forum.org/download/af-pnni-
0055.001.pdf>.
[RFC3626] Clausen, T., Ed. and P. Jacquet, Ed., "Optimized Link
State Routing Protocol (OLSR)", RFC 3626,
DOI 10.17487/RFC3626, October 2003,
<https://www.rfc-editor.org/info/rfc3626>.
[RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
Border Gateway Protocol 4 (BGP-4)", RFC 4271,
DOI 10.17487/RFC4271, January 2006,
<https://www.rfc-editor.org/info/rfc4271>.
[RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch,
"Network Time Protocol Version 4: Protocol and Algorithms
Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010,
<https://www.rfc-editor.org/info/rfc5905>.
[RFC8200] Deering, S. and R. Hinden, "Internet Protocol, Version 6
(IPv6) Specification", STD 86, RFC 8200,
DOI 10.17487/RFC8200, July 2017,
<https://www.rfc-editor.org/info/rfc8200>.
[RFC8505] Thubert, P., Ed., Nordmark, E., Chakrabarti, S., and C.
Perkins, "Registration Extensions for IPv6 over Low-Power
Wireless Personal Area Network (6LoWPAN) Neighbor
Discovery", RFC 8505, DOI 10.17487/RFC8505, November 2018,
<https://www.rfc-editor.org/info/rfc8505>.
[RFC8928] Thubert, P., Ed., Sarikaya, B., Sethi, M., and R. Struik,
"Address-Protected Neighbor Discovery for Low-Power and
Lossy Networks", RFC 8928, DOI 10.17487/RFC8928, November
2020, <https://www.rfc-editor.org/info/rfc8928>.Acknowledgments 🫶
The authors wanna give a huge shoutout to Jaroslaw Kowalczyk, Alvaro Retana, Jim Guichard, and Jeffrey Zhang for providing invaluable concepts and content for this document 💜.
Contributors 🌸
The following people contributed substantially to the content of this
document and should be considered coauthors:
Jordan Head
Juniper Networks
Email: jhead@juniper.net
Tom Verhaeg
Juniper Networks
Email: tverhaeg@juniper.netAuthors' Addresses 💖
okay bestie, meet the squad behind this one 🫶 the editor is Yuehua Wei, who's giving main character energy as editor 👑
Yuehua Wei (editor) 🌸
Yuehua Wei (editor)
ZTE Corporation
No.50, Software Avenue
Nanjing
210012
China
Email: wei.yuehua@zte.com.cnnext up is Zheng (Sandy) Zhang, also from ZTE Corporation in the same Nanjing office 🦋
Zheng (Sandy) Zhang 💜
Zheng (Sandy) Zhang
ZTE Corporation
No.50, Software Avenue
Nanjing
210012
China
Email: zhang.zheng@zte.com.cnthen Dmitry Afanasiev, listed with Yandex and an email but no postal address 🙈
Dmitry Afanasiev 🧚
Dmitry Afanasiev
Yandex
Email: fl0w@yandex-team.ruPascal Thubert is listed as an Individual based in France, which is so iconic 🇫🇷✨
Pascal Thubert 🍓
Pascal Thubert
Individual
France
Email: pascal.thubert@gmail.comand last but definitely not least, Tony Przygienda at Juniper Networks in Sunnyvale, CA, United States of America 🌈
Tony Przygienda 🎀
Tony Przygienda
Juniper Networks
1194 N. Mathilda Ave
Sunnyvale, CA 94089
United States of America
Email: prz@juniper.netthat's the whole crew 💅 five people, two ZTE teammates, one Yandex, one solo France rep, and one Juniper legend 🔥
📦 where this page comes from
- ietf: rfc9696 2025-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms