RFC 9692
RIFT: Routing in Fat Trees
RFC 9692: RIFT: Routing in Fat Trees 🦋
Part 1 of 39
Okay bestie, this one is literally a Standards Track document from the IETF, published in April 2025 with ISSN 2070-1721. The editors are T. Przygienda (Ed.) and J. Head (Ed., Juniper Networks). The other authors are A. Sharma (Hudson River Trading), P. Thubert, B. Rijsman (Individual), and D. Afanasiev (Yandex). This header doesn't list anything it obsoletes or updates, so there's no drama there ✨
Abstract
okay so this doc defines a specialized, dynamic routing protocol for Clos, fat tree, and their variants 🌸 These topologies started out inside crossbar interconnects, and that's how they ended up in router and switch backplanes. But the characteristics are so good that they're also perfect for building IP fabrics 😭 The protocol is designed to keep control plane state as small as possible so it can handle really huge substrates, and to cut down on configuration and operational complexity so these topologies are way easier to deploy 💅
Status of This Memo
this is an Internet Standards Track document 🎀
It's a product of the Internet Engineering Task Force (IETF), so it represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). If you want more about Internet Standards, check out Section 2 of RFC 7841 🫶
For the current status, any errata, and how to send feedback, go to https://www.rfc-editor.org/info/rfc9692 💜
Copyright Notice
Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document 🧚 Any Code Components pulled out of this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're provided without warranty, as described in the Revised BSD License ⭐
Table of Contents
1. Introduction
1.1. Requirements Language
2. A Reader's Digest
3. Reference Frame
3.1. Terminology
3.2. Topology
4. RIFT: Routing in Fat Trees
5. Overview
5.1. Properties
5.2. Generalized Topology View
5.2.1. Terminology and Glossary
5.2.2. Clos as Crossed, Stacked Crossbars
5.3. Fallen Leaf Problem
5.4. Discovering Fallen Leaves
5.5. Addressing the Fallen Leaves Problem
6. Specification
6.1. Transport
6.2. Link (Neighbor) Discovery (LIE Exchange)
6.2.1. LIE Finite State Machine
6.3. Topology Exchange (TIE Exchange)
6.3.1. Topology Information Elements
6.3.2. Southbound and Northbound TIE Representation
6.3.3. Flooding
6.3.4. TIE Flooding Scopes
6.3.5. RAIN: RIFT Adjacency Inrush Notification
6.3.6. Initial and Periodic Database Synchronization
6.3.7. Purging and Rollovers
6.3.8. Southbound Default Route Origination
6.3.9. Northbound TIE Flooding Reduction
6.3.10. Special Considerations
6.4. Reachability Computation
6.4.1. Northbound Reachability SPF
6.4.2. Southbound Reachability SPF
6.4.3. East-West Forwarding Within a Non-ToF Level
6.4.4. East-West Links Within a ToF Level
6.5. Automatic Disaggregation on Link & Node Failures
6.5.1. Positive, Non-Transitive Disaggregation
6.5.2. Negative, Transitive Disaggregation for Fallen Leaves
6.6. Attaching Prefixes
6.7. Optional Zero Touch Provisioning (RIFT ZTP)
6.7.1. Terminology
6.7.2. Automatic System ID Selection
6.7.3. Generic Fabric Example
6.7.4. Level Determination Procedure
6.7.5. RIFT ZTP FSM
6.7.6. Resulting Topologies
6.8. Further Mechanisms
6.8.1. Route Preferences
6.8.2. Overload Bit
6.8.3. Optimized Route Computation on Leaves
6.8.4. Mobility
6.8.5. Key/Value (KV) Store
6.8.6. Interactions with BFD
6.8.7. Fabric Bandwidth Balancing
6.8.8. Label Binding
6.8.9. L2L Procedures
6.8.10. Address Family and Multi-Topology Considerations
6.8.11. One-Hop Healing of Levels with East-West Links
6.9. Security
6.9.1. Security Model
6.9.2. Security Mechanisms
6.9.3. Security Envelope
6.9.4. Weak Nonces
6.9.5. Lifetime
6.9.6. Security Association Changes
7. Information Elements Schema
7.1. Backwards-Compatible Extension of Schema
7.2. common.thrift
7.3. encoding.thrift
8. Further Details on Implementation
8.1. Considerations for Leaf-Only Implementation
8.2. Considerations for Spine Implementation
9. Security Considerations
9.1. General
9.2. Time to Live and Hop Limit Values
9.3. Malformed Packets
9.4. RIFT ZTP
9.5. Lifetime
9.6. Packet Number
9.7. Outer Fingerprint Attacks
9.8. TIE Origin Fingerprint DoS Attacks
9.9. Host Implementations
9.9.1. IPv4 Broadcast and IPv6 All-Routers Multicast
Implementations
10. IANA Considerations
10.1. Multicast and Port Numbers
10.2. Registry for RIFT Security Algorithms
10.3. Registries with Assigned Values for Schema Values
10.3.1. RIFTVersions Registry
10.3.2. RIFTCommonAddressFamilyType Registry
10.3.3. RIFTCommonHierarchyIndications Registry
10.3.4. RIFTCommonIEEE8021ASTimeStampType Registry
10.3.5. RIFTCommonIPAddressType Registry
10.3.6. RIFTCommonIPPrefixType Registry
10.3.7. RIFTCommonIPv4PrefixType Registry
10.3.8. RIFTCommonIPv6PrefixType Registry
10.3.9. RIFTCommonKVTypes Registry
10.3.10. RIFTCommonPrefixSequenceType Registry
10.3.11. RIFTCommonRouteType Registry
10.3.12. RIFTCommonTIETypeType Registry
10.3.13. RIFTCommonTieDirectionType Registry
10.3.14. RIFTEncodingCommunity Registry
10.3.15. RIFTEncodingKeyValueTIEElement Registry
10.3.16. RIFTEncodingKeyValueTIEElementContent Registry
10.3.17. RIFTEncodingLIEPacket Registry
10.3.18. RIFTEncodingLinkCapabilities Registry
10.3.19. RIFTEncodingLinkIDPair Registry
10.3.20. RIFTEncodingNeighbor Registry
10.3.21. RIFTEncodingNodeCapabilities Registry
10.3.22. RIFTEncodingNodeFlags Registry
10.3.23. RIFTEncodingNodeNeighborsTIEElement Registry
10.3.24. RIFTEncodingNodeTIEElement Registry
10.3.25. RIFTEncodingPacketContent Registry
10.3.26. RIFTEncodingPacketHeader Registry
10.3.27. RIFTEncodingPrefixAttributes Registry
10.3.28. RIFTEncodingPrefixTIEElement Registry
10.3.29. RIFTEncodingProtocolPacket Registry
10.3.30. RIFTEncodingTIDEPacket Registry
10.3.31. RIFTEncodingTIEElement Registry
10.3.32. RIFTEncodingTIEHeader Registry
10.3.33. RIFTEncodingTIEHeaderWithLifeTime Registry
10.3.34. RIFTEncodingTIEID Registry
10.3.35. RIFTEncodingTIEPacket Registry
10.3.36. RIFTEncodingTIREPacket Registry
11. References
11.1. Normative References
11.2. Informative References
Appendix A. Sequence Number Binary Arithmetic
Appendix B. Examples
B.1. Normal Operation
B.2. Leaf Link Failure
B.3. Partitioned Fabric
B.4. Northbound Partitioned Router and Optional East-West Links
Acknowledgments
Contributors
Authors' Addresses1. Introduction ✨
okay so Clos [CLOS] topologies are literally everywhere rn 💖. The big reason is that everyone's going all in on centralized data centers, the place where most computation and storage is gonna live going forward 🏢✨. These networks get called a fat tree / network in modern IP fabric talk [VAHDAT08], which is a cute nod to the original Fat Tree idea [FATTREE]. Generally speaking (ignoring stuff like horizontal shortcuts), they're all versions of a structured design that's basically a ranked lattice 🧩. The least upper bound is the "top of the fabric," and links nearer the top can be "fatter" so you get non-blocking bisectional capacity 💅.
Lots of people building these IP fabrics want a protocol that sets itself up automatically and handles failures and misconfigurations with barely any human help 🫶. That would make local IP fabric bandwidth feel like a "standard component," so you could provision it way faster and run it for way less money than today. Basically, the same way compute or storage is already consumed 🍓.
Now, Routing in Fat Trees (RIFT) tackles these challenges 🦋. It doesn't just tweak a link-state approach (distributed computation) or a distance-vector approach (diffused computation). It mixes both, and the short version is "link-state towards the spines" and "distance vector towards the leaves." So the "bottom" levels flood their link-state info in the "northern" direction, while each node normally generates a "default route" and floods it in the "southern" direction. This style naturally gives you super desirable address aggregation 😍. The catch is that aggregation can drop traffic when there's misconfiguration or while failures are still being sorted out 😭. It could also cause persistent network partitioning, and that needs some adequate mechanism to fix it. RIFT's answer is in Section 6.5: automatic, sufficient disaggregation of prefixes when links or nodes fail.
The protocol also gives you:
- optional fully automated construction of fat tree topologies based on detection of links without any configuration (Section 6.7), while still allowing conventional configuration methods or an arbitrary mix of both,
- the minimum amount of routing state held by nodes,
- automatic pruning and load balancing of topology flooding exchanges over a sufficient subset of links (Section 6.3.9),
- automatic address aggregation (Section 6.3.8) and consequently automatic disaggregation (Section 6.5) of prefixes on link and node failures to prevent traffic loss and suboptimal routing,
- loop-free non-ECMP forwarding due to its inherent valley-free nature,
- fast mobility (Section 6.8.4),
- rebalancing of traffic towards the spines based on bandwidth available (Section 6.8.7.1), and finally
- mechanisms to synchronize a limited key-value datastore (Section 6.8.5.1) that can be used after protocol convergence to, e.g., bootstrap higher levels of functionality on nodes.
Figure 1 shows a simplified, conceptual view of a RIFT fabric with its routing tables and topology databases, using IPv4 as the address family 🌈. The top of the fabric's link-state database holds info about the nodes below it and the routes to them. When you look at Figure 1, /32 notation is each node's IPv4 loopback address (so A/32 is node A's loopback, and so on), and 0/0 means a default IPv4 route. The first row of database info covers the nodes where full topology info is available. The second row says partial info about other nodes in the same level is also available too. That info is needed for certain algorithms that the protocol relies on to work correctly 🧚. At the "bottom" (aka the leaves) of the fabric, the topology is basically empty, and under normal conditions the leaves hold a load-balanced default route to the next level 🌸.
The rest of this document fills in the protocol specification details 📝.
[A,B,C,D]
[E]
+---------+ +---------+ A/32 @ [C,D]
| E | | F | B/32 @ [C,D]
+-+-----+-+ +-+-----+-+ C/32 @ C
| | | | D/32 @ D
| | | |
| | +--+ |
| | | |
| +---------)--+ |
| | | |
| | | |
| +---------+ | |
| | | |
[A,B] +-+-----+-+ +-+-----+-+ [A,B]
[D] | C | | D | [C]
+-+-----+-+ +-+-----+-+
0/0 @ [E,F] | | | | 0/0 @ [E,F]
A/32 @ [A] | | | | A/32 @ A
B/32 @ [B] | | +--+ | B/32 @ B
| | | |
| +---------)--+ |
| | | |
| +---------+ | |
| | | |
+-+-----+-+ +-+-----+-+
0/0 @ [C,D] | A | | B | 0/0 @ [C,D]
+---------+ +---------+Figure 1: RIFT Information Distribution
1.1. Requirements Language
Here's the key word bit 🫶. The words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this doc get read the way BCP 14 [RFC2119] [RFC8174] describes. But that only happens when, and only when, they show up in ALL CAPS, exactly like this 💜.
2. A Reader's Digest ✨
okay so this section is a guided tour of the whole doc, sorted by how deep you wanna go 🗺️💖 the authors say to read the HTML or PDF versions bc plain text can't show complex figures very well 🎀
the "Terminology" (Section 3.1) section is your bestie for looking stuff up as you go 📖
the directions ("top", "bottom", etc.) from Section 1 are a HUGE deal 🔝🔻 RIFT needs a topology with a clear sense of top and bottom so it can build a sorted topology. Clos, fat tree, and similar structures are a perfect fit for that 🌸 we'll mention later where RIFT loosens these rules a bit ✨
a bunch of the images are labeled "northern view" or "southern view" to show the perspective 📸 "northern view" means "from the top of the fabric looking down" 👀⬇️ and "southern view" means "from the bottom looking up" 👀⬆️
ops folks and implementors both need to decide if multi-plane IP fabrics are for them 💅 Section 3.2 shows a single-plane example in Figure 2 and a multi-plane fabric in Figure 3 🦋 multi-plane fabrics need extra RIFT concepts (like negative disaggregation in Section 6.5.2) that you don't need for single-plane only fabrics 🧩 "Overview" (Section 5) and "Generalized Topology View" (Section 5.2) give you enough context to figure out if multi-plane is your thing 🔮 then "Fallen Leaf Problem" (Section 5.3) plus Sections 5.4 and 5.5 cover extra stuff that's specific to multi-plane 💜
the fundamental protocol concepts start in "Specification" (Section 6) 📚 but some subsections only matter if you're actually implementing the protocol 🛠️ the protocol transport (Section 6.1) is a MUST-read 😭 wait no, not capitals, lol: it's super important for two reasons 🎀 first, it introduces RIFT's packet format content as a normative Thrift [thrift] model in Section 7.3, and that gets carried in a security envelope described in Section 6.9.3 📦 second, the Thrift model is a prerequisite for understanding RIFT's built-in security features in both "Security" (Section 6.9) and "Security Considerations" (Section 9) 🔐 the normative schema for the Thrift model is in Sections 7.2 and 7.3 🧾 and while you don't need to understand Thrift [thrift] and the model in depth unless you're implementing RIFT, they can be super useful for other readers too 🌈
if you're implementing RIFT for multi-plane topologies, read all of Section 6 together with those Thrift schemas 💖 sections that don't matter for single-plane implementations get called out later in this section 🧚
everyone implementing the protocol should give the Link Information Element (LIE) definitions (Section 6.2) extra attention 👀 they cover basic neighbor discovery and adjacency formation, and they also give context for RIFT's optional Zero Touch Provisioning (ZTP) (Section 6.7) and miscabling detection 🪄 those let RIFT automatically detect and build the underlay topology with basically zero configuration 🤯 all of that is detailed in Section 6.7 ✨
for everyone else, the next sections go deeper on the core properties and some extra perks of RIFT 💅 like link-state packet formats, efficient flooding, synchronization, loop-free path computation, and link-state database maintenance (see Sections 6.3, 6.3.2, 6.3.3, 6.3.4, 6.3.6, 6.3.7, 6.3.8, 6.4, 6.4.1, 6.4.2, 6.4.3, and 6.4.4) 🌸 RIFT can also do weighted unequal-cost load balancing of traffic across all available links ⚖️ that's outlined in Section 6.8.7 with an example 🍓
Section 6.5 explains the single-plane vs. multi-plane requirement in more detail 🦋 if you're into single-plane fabrics, only Section 6.5.1 is needed 🎀 if you're into multi-plane, Sections 6.5.2, 6.5.2.1, 6.5.2.2, and 6.5.2.3 are also a must 😭 Section 6.6 is especially important for multi-plane folks 🔥 it shows how the Routing Information Base (RIB) and Forwarding Information Base (FIB) get built through the disaggregation mechanisms, and how they stop bad routing decisions that cause traffic loss in both single-plane and multi-plane topologies 🛡️
Appendix B has a set of detailed examples showing how RIFT keeps the impact of failures limited to just the nodes that need to know 🧠 it's a great way to lock in RIFT's core concepts 💜
last but not least, RIFT has other optional capabilities 🌈 one is the key-value datastore, which lets RIFT advertise data after convergence to bootstrap higher levels of functionality (e.g., operational telemetry) 📊 those are covered in Section 6.8 ✨
more RIFT info lives in the "RIFT Applicability" [RFC9696] document, which covers alternate topologies RIFT can run on, use cases where it fits, and operational considerations that go with this doc 🧩 "RIFT Day One" [DayOne] covers practical details of existing RIFT implementations and deployments 🚀
3. Reference Frame 🖼️
3.1. Terminology 📖
this section is where all the terms used in this doc are defined, bestie 🫶
Bandwidth Adjusted Distance (BAD): each RIFT node can work out how much northbound bandwidth it has toward a node compared to other nodes at the same level 📶 then it can change the route distance to match, so the lower level can adjust its load balancing toward spines ⚖️
Bidirectional Adjacency: a bidirectional adjacency is one where nodes on both sides advertised it in their Node TIEs with the correct levels and System IDs 🤝 bidirectionality is what different algorithms use to check whether the link should be included ✅
Bow-tying: in fully converged IP fabrics, traffic normally takes the shortest route by hop count to its destination (e.g., leaf, spine, leaf) 🛣️ but some failure scenarios with partial routing info make nodes lose the downstream reachability they need 😱 that forces traffic to go up to higher levels and then turn south again through a different route to get reachable (e.g., leaf, spine-1, superspine, spine-2, leaf) 🌀 that's the bow-tie shape 🎀
Clos / fat tree: this doc uses "Clos" and "fat tree" interchangeably, and both always mean a folded spine-and-leaf topology 🧵 it can have multiple Points of Delivery (PoDs) and one or multiple Top of Fabric (ToF) planes 🌐 modifications like L2L shortcuts and multi-level shortcuts are possible, and the doc describes them further 🦋
Cost: a natural number with no unit, tied to a single entity 🔢 cost is a monoid under addition ➕ a cost can belong to a single link or prefix, or it can stand for the sum of costs (distance) of the links along the path between two nodes 🧮
Crossbar: the physical arrangement of ports in a switching matrix 🔀 it doesn't imply any scheduling or buffering rules on top of that 💅
Directed Acyclic Graph (DAG): a finite directed graph with no directed cycles (loops) 🔁❌ if you think of all the links in a Clos as pointing either all toward the top or all toward the bottom, each of those two graphs is a DAG 🌈
Disaggregation: the process where a node decides to advertise more specific prefixes southward 📣 it can do this positively to attract the matching traffic, or negatively to repel it 🙅♀️ disaggregation is done to prevent traffic loss and suboptimal routing to the more specific prefixes 💖
Distance: the sum of costs (bound by the infinite cost constant) between two nodes 📏 distance mainly shows how far apart two entities are, and it can be reused as cost in another context 🔄
East-West (E-W) Link: a link between two nodes at the same level ↔️ East-West links are normally not part of Clos or fat tree topologies 🙈
Flood Repeater (FR): a node can pick one or more northbound neighbor nodes to be flood repeaters 📡 the flood repeaters are in charge of flooding northbound TIEs further north 🚀 the doc sometimes calls them flood leaders too 👑
Folded Spine-and-Leaf: when the Clos fabric's input and output stages are equivalent, the fabric can be "folded" into a "superspine" or top, which this doc calls the ToF 🪞 so cute 😭
Interface: a layer 3 entity that RIFT control packets get exchanged over 📨
Key Value (KV) TIE: a TIE that carries a set of key value pairs [DYNAMO] 🗝️ it can be used to distribute non-topology-related information inside the protocol 🎁
Leaf-to-Leaf (L2L) Shortcuts: East-West links at leaf level have to be told apart from East-West links at other levels 👯♀️
Leaf: a node with no southbound adjacencies 🍃 level 0 means a leaf in RIFT, but a leaf doesn't have to be level 0 🌱
Level: Clos and fat tree networks are topologically partially ordered graphs, and "level" means the set of nodes at the same height in one of them 🪜 nodes at the top level (i.e., ToF) have the highest value, and the count goes down to the bottom level (i.e., leaf) with the lowest value ⬇️ a node has links to nodes one level down and/or one level up ↕️ sometimes a node can link to other nodes at its own level 🤪 a leaf node can also link to nodes several levels higher 🚀 in RIFT, level 0 always means a node is a leaf, but a leaf doesn't have to be level 0 🌈 level values can be set manually or automatically, as described in Section 6.7 🧚
here's the footnote, bestie 🤓
| As a final footnote: Clos terminology often uses the concept
| of "stage", but due to the folded nature of the fat tree, it
| is not used from this point on to prevent misunderstandings.LIE: short for "Link Information Element" 💌 it gets exchanged on all of the system's RIFT links to form _ThreeWay_ adjacencies, and it carries the info used for RIFT Zero Touch Provisioning (ZTP) of levels 🎀
Metric: used interchangeably with "cost" 🔁
Neighbor: once a _ThreeWay_ adjacency has formed, a neighborship relationship holds the neighbor's properties 🧑🤝🧑 you can form multiple adjacencies to a remote node over parallel point-to-point interfaces, but those adjacencies do not share a neighbor structure 🙅♀️ so saying "neighbor" is the same thing as saying "a _ThreeWay_ adjacency" 💜
Okay so, the Node TIE 💖 is short for "Node Topology Information Element," and it holds every adjacency the node discovered plus info about the node itself. ✨ Don't mix it up with a North TIE! "node" tells you the type of TIE, not its direction, so you get North Node TIEs and South Node TIEs. 💅
North SPF (N-SPF): this is a reachability calculation that goes northbound, like SPF that only uses South Node TIEs. 🦋 It normally moves only a single hop and installs default routes. 🌸
Northbound Link: a link to a node one level up, or in other words, one level further north. ⭐
Northbound Representation: the subset of topology info that gets flooded toward the higher levels of the fabric. 🎀
Overloaded: this applies to a node advertising the overload attribute as set. 🙈 The overload attribute is carried in the NodeFlags object of the encoding schema.
Point of Delivery (PoD): a self-contained vertical slice (or subset) of a Clos or fat tree network, usually with only level 0 and level 1 nodes. 🧚 A node in a PoD talks to nodes in other PoDs through the ToF nodes. PoDs get numbers so you can tell them apart, and PoD value 0 (defined later in the encoding schema as common.default_pod) means "undefined" or "any" PoD. 💜
Prefix TIE: short for "Prefix Topology Information Element." 🍓 It contains all prefixes directly attached to this node in the case of a North TIE, and the necessary default routes the node advertises southbound in the case of a South TIE.
Radix: the radix of a switch is the number of switching ports it provides. It's sometimes called "fanout" too. 🌈
Routing on the Host (RotH): a modern data center architecture variant where servers/leaves are multihomed and, consequently, take part in routing. 👑
Security Envelope: RIFT packets get flooded inside an authenticated security envelope, which optionally protects the integrity of the information a node accepts if any of the mechanisms in Section 10.2 are used. 🔥 This is described further in Section 6.9.3.
Shortest Path First (SPF): a well-known graph algorithm attributed to Dijkstra [DIJKSTRA] that builds a tree of shortest paths from a source to destinations on the graph. 🧠 We use the SPF acronym because it's familiar as a general term for the node reachability calculations RIFT can use to finally calculate routes. Dijkstra's algorithm is one possible option. ✨
South Reflection: often shortened to just "reflection." 🪞 It's a mechanism where South Node TIEs get "reflected" from the level south back up north, so nodes in the same level without E-W links can still know about each other's node Topology Information Elements (TIEs). 💖
South SPF (S-SPF): a reachability calculation that goes southbound, like SPF that only uses North Node TIEs. 🌸
South/Southbound and North/Northbound (Direction): when describing protocol elements and procedures, the compass directions get used in different situations. "lower," "south," and "southbound" all mean moving toward the bottom of the Clos or fat tree network, and "higher," "north," and "northbound" mean moving toward the top. 🧭
Southbound Link: a link to a node one level down, or in other words, one level further south. 🍓
Southbound Representation: the subset of topology info that gets sent toward a lower level. 🦋
Spine: any nodes north of leaves and south of ToF nodes. 🌲 There can be multiple layers of spines in a PoD. 💅
Superspine, Aggregation/Spine, and Edge/Leaf Switches: these are typical level names in 5 stages folded Clos for levels 2, 1, and 0, respectively (counting up from the bottom). 🎀 We normalize this language and talk about ToF, Top-of-Pod (ToP), and leaves instead. ✨
System ID: RIFT nodes identify themselves with a unique network-wide number when they try to build adjacencies or describe their topology. 🫶 System IDs can be auto-derived or configured.
ThreeWay Adjacency: RIFT tries to form a unique adjacency between two nodes over a point-to-point interface and exchanges local configuration and the necessary RIFT ZTP information. 🤝 An adjacency only gets advertised in Node TIEs and used for computations after it reaches the ThreeWay state, i.e., both routers have reflected each other in LIEs, including the relevant security information. 🔐 Still, LIEs sent before the ThreeWay state is reached may already carry information related to RIFT ZTP. 🌟
TIDE: the Topology Information Description Element carries descriptors of the TIEs stored in the node. 📋
TIE: short for "Topology Information Element." 💜 TIEs get exchanged between RIFT nodes to describe parts of a network, such as links and address prefixes. A TIE always has a direction and a type. North TIEs (sometimes abbreviated as N-TIEs) are used when dealing with TIEs in the northbound representation, and South-TIEs (sometimes abbreviated as S-TIEs) are used for the southbound equivalent. TIEs come in different types, such as node and prefix TIEs. 🌈
TIEDB: the database holding the newest versions of all TIE headers (and the corresponding TIE content, if it's available). 🗄️
TIRE: the Topology Information Request Element carries a set of TIDE descriptors. 🙌 It can both confirm received TIEs and request missing ones.
Top of Fabric (ToF): the set of nodes that provide inter-PoD communication and have no northbound adjacencies, i.e., they sit at the "very top" of the fabric. 👑 ToF nodes don't belong to any PoD, and they're assigned the common.default_pod PoD value to indicate the equivalent of "any" PoD. 💖
Top of PoD (ToP): the set of nodes that provide intra-PoD communication and have northbound adjacencies outside of the PoD, i.e., they're at the "top" of the PoD. 🌸
ToF Plane or Partition: in large fabrics, ToF switches may not have enough ports to aggregate all the switches south of them, so the ToF gets "split" into multiple independent planes. 🍰 Section 5.2 explains the concept in more detail. A plane is a subset of ToF nodes that are aware of each other through south reflection or E-W links. 🫧
Valid LIE: LIEs go through different checks to determine their validity. 🔍 The term "valid LIE" describes a LIE that can be used to form or maintain an adjacency. How much checking happens depends on the Finite State Machine (FSM) involved and its state. A "minimally valid LIE" is a LIE that passes the checks necessary on any FSM in any state. A "ThreeWay valid LIE" is a LIE that successfully went through further checks with a LIE FSM in the ThreeWay state. 💅 A minimally valid LIE is a subcategory of a ThreeWay valid LIE. ✨
RIFT Zero Touch Provisioning (abbreviated as RIFT ZTP or just ZTP): an optional RIFT mechanism that allows the automatic derivation of node levels based on minimum configuration, as detailed in Section 6.7. 🪄 That minimum configuration consists solely of ToFs being configured as such. RIFT ZTP also contains a recommendation for automatic, collision-free derivation of the System ID. 🎀
Additionally, when the specification refers to elements of packet encoding or the constants provided in Section 7, a special emphasis is used, e.g., invalid_distance. 🌟 The same convention is used when referring to finite state machine states or events outside the context of the machine itself, e.g., OneWay. 💜
3.2. Topology 🗺️
^ N +--------+ +--------+
Level 2 | |ToF 21| |ToF 22|
W <-*-> E ++-+--+-++ ++-+--+-++
| | | | | | | | |
S v P111/2 P121/2 | | | |
MHP MHP | | | |
^ ^ ^ ^ | | | |
| | | | | | | |
+--------------+ | +-----------+ | | | +---------------+
| | | | | | | |
| +-----------------------------+ | | ^
| | | | | | | All
0/0 0/0 0/0 +-----------------------------+ TIEs
v v v | | | | |
| | +-+ +<-0/0----------+ | |
| | | | | | | |
+-+----++ Optional +-+----++ ++----+-+ ++-----++
Level 1 | | E/W Link | | | | | |
|Spin111+----------+Spin112| |Spin121| |Spin122|
+-+---+-+ ++----+-+ +-+---+-+ ++---+--+
| | | | | | | |
| +---0/0--->-----+ 0/0 | +----------------+ |
0/0 | | | | | | |
| +---<-0/0-----+ | v | +--------------+ | |
v | | | | | | |
+-+---+-+ +--+--+-+ +-+---+-+ +---+-+-+
Level 0 | | L2L | | | | | |
|Leaf111+~~~~~~~~~~+Leaf112| |Leaf121| |Leaf122|
+-+-----+ +-+---+-+ +--+--+-+ +-+-----+
+ + \ / + +
Prefix111 Prefix112 \ / Prefix121 Prefix122
Multi-homed
Prefix
+---------- PoD 1 ---------+ +---------- PoD 2 ---------+Figure 2: A Three-Level Spine-and-Leaf Topology 🌲
The topology in Figure 2 is referred to in all further considerations. ✨ This figure depicts a generic "single-plane fat tree," and the concepts explained using three levels apply by induction to further levels and higher degrees of connectivity. 🧠
(Artwork only available as SVG: see
https://www.rfc-editor.org/rfc/rfc9692.html)Figure 3: Topology with Multiple Planes 🦋
Further, this document will also deal with designs that provide only sparser connectivity and "partitioned spines," as shown in Figure 3 and explained further in Section 5.2. 🫶
4. RIFT: Routing in Fat Trees 🌈
The remainder of this document presents the detailed specification of the RIFT protocol. In the most abstract terms, it has many properties of a modified link-state protocol when distributing information northbound, and a distance-vector protocol when distributing information southbound. 🤯 While this is an unusual combination, it does quite naturally exhibit desired properties. 👑
5. Overview ✨
5.1. Properties
okay so the most iconic thing about RIFT 💅 is that it only floods link-state info northbound, so each level ends up with the full topology of the levels south of it 🧚. Link-state info is mostly NOT flooded east-west, and it doesn't go back south either (with a few exceptions 👀). South reflection is the exception we unpack in Section 6.5.1, and east-west flooding at the ToF level in multi-plane fabrics gets its own spotlight in Section 5.2 ✨. Going south, the routing info you actually need (usually just a default route, see Section 6.3.8) only travels one hop down 🍓. Then those nodes make their own routing info and flood it south, so nobody has to build a fresh update for every single adjacency. Rest of the East-West story comes later in the doc 🫶.
Those flow rules make RIFT anisotropic 🌈. That's a fancy way of saying the info isn't spread "evenly" or "clumped" everywhere, it gets summarized along the north-south gradient. It's also smooth, because nodes don't get the same info from multiple directions at the same time 🦋. Normally, if you accept the same reachability on every link without understanding what it means topologically, you have to tie-break with some distance function 😭. That tie-breaking ends up with hop-by-hop forwarding along shortest paths only. RIFT doesn't do that. Under normal conditions it never needs to tie-break the same reachability coming from multiple directions, because its rule of always preferring the south forwarding direction leads to valley-free [VFR] forwarding ✨. In short, valley-free paths let a packet heading north turn around and head south, while horizontal links can be crossed in the northbound phase at most once 🎀. Those rules guarantee loop-free forwarding, so the fabric can use all of its feasible paths 🔥. That's a super desirable property if you want to squeeze every bit of available bandwidth out of the network 💜.
To handle the "northern" and "southern" split of info, the link state database (LSDB) gets partitioned into "north representation" and "south representation" Topology Information Elements (TIEs) 🌸. Simply put, North TIEs hold a link-state topology description of the lower levels, and South TIEs just carry a node description of the level above plus default routes pointing north 💖. This is a super simplified picture. The next sections refine it bit by bit while they introduce the protocol procedures and state machines ⭐.
5.2. Generalized Topology View
okay this section and Section 6.5.2 are all about multi-plane fabrics 🦋. That's different from single-plane designs, where every ToF node is topologically equal and initially connected to all the switches one level below it 💅.
Multi-plane designs are basically a multidimensional switching matrix 🌈. To make that easier to picture, this doc introduces a way to draw the connectivity in two-dimensional pictures ✨. You can lean on the fact that what we're looking at is really stacked crossbar fabrics, where the ports line up "on top of each other" in a neat, regular pattern 🧚.
Quick warning, bestie 🚨: the language people use to describe Clos variations, especially for multi-plane designs, is all over the place depending on the source 😭. This description follows the terminology from Section 3.1, and you need that terminology to follow the rest of this section properly 🫶.
5.2.1. Terminology and Glossary
This section lays out the terms and abbreviations used in the rest of the text 📖. The glossary might be a bit confusing on the first read, but the following sections introduce each term in its proper context 💜.
P: Number of PoDs in a topology 🌸.
S: Number of ToF nodes in a topology 🧚.
K: To keep the visuals and notation simple, we assume the switches are symmetrical, meaning they have an equal number of ports pointing northbound and southbound 🎀. With that assumption, K is half the radix of a symmetrical switch, so the switch has K ports pointing north and K ports pointing south 🔥. K_LEAF (K of a leaf) is both the number of access ports on a leaf node and the maximum number of planes in the fabric 💅. K_TOP (K of a ToP) is the number of leaves in the PoD, and also the number of ports pointing north from a ToP node toward a higher spine level, which means it's the number of ToF nodes in a plane 🦋.
ToF Plane: A set of ToFs that are aware of each other through south reflection 🌈. Planes get named with capital letters, like plane A ✨.
N: Number of independent ToF planes in a topology 💖.
R: A redundancy factor, meaning the number of ToP nodes in a PoD that connect to a ToF plane 🍓. In a single-plane design, R equals K_LEAF ⭐.
Fallen Leaf: A fallen leaf in a plane Z is a switch that has lost ALL of its northbound connectivity to Z 😭💀.
5.2.2. Clos as Crossed, Stacked Crossbars
The typical topology RIFT is built for has P PoDs, all connected together by S ToF nodes 🧚. A PoD node has 2K ports 🎀. From here on, half of them (K=Radix/2) connect host devices from the south, and the other half connect to interleaved PoD top-level switches to the north 🫶. You can pick a different K ratio without losing generality when port speeds differ or the fabric is oversubscribed 🔥. Still, K=Radix/2 makes things way more readable, because every intermediate node then has as many ports facing north as south 💜. So we draw a node as a schematic with ports "sticking out" to its north and south, instead of the usual real-world front faceplate designs 🌸.
Figure 4 shows a leaf node as seen from the north, meaning the ports that connect northbound 👀. Since there's no better symbol, the document uses "o" to represent a single port 💖. In this example, K_LEAF has 6 ports ✨. Heads up: the number of PoDs isn't tied to the radix, unless the ToF nodes happen to be the same as the PoD nodes in a particular deployment 🦋.
Top View
+---+
| |
| o | e.g., Radix = 12, K_LEAF = 6
| |
| o |
| | -------------------------
| o <------ Physical Port (Ethernet) ----+
| | ------------------------- |
| o | |
| | |
| o | |
| | |
| o | |
| | |
+---+ v
|| || || || || || ||
+----+ +------------------------------------------------+
| | | |
+----+ +------------------------------------------------+
|| || || || || || ||
Side Views
Figure 4: A Leaf Node, K_LEAF=6A PoD's top node can have a different radix than the leaf node 🌈. But most of the time, the same type of node gets used for both, which makes a square (K*K) 🎀. In the general case, you could also consider switches at the top of the PoD whose K_TOP southern ports aren't necessarily equal to K_LEAF 💅. For example, the diagrams below use a 6-port K_LEAF and an 8-port K_TOP ✨. To form a crossbar, you need K_TOP leaf nodes, as Figure 5 shows 🧚.
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
| | | | | | | | | | | | | | | |
| O | | O | | O | | O | | O | | O | | O | | O |
| | | | | | | | | | | | | | | |
| O | | O | | O | | O | | O | | O | | O | | O |
| | | | | | | | | | | | | | | |
| O | | O | | O | | O | | O | | O | | O | | O |
| | | | | | | | | | | | | | | |
| O | | O | | O | | O | | O | | O | | O | | O |
| | | | | | | | | | | | | | | |
| O | | O | | O | | O | | O | | O | | O | | O |
| | | | | | | | | | | | | | | |
| O | | O | | O | | O | | O | | O | | O | | O |
| | | | | | | | | | | | | | | |
| O | | O | | O | | O | | O | | O | | O | | O |
| | | | | | | | | | | | | | | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
Figure 5: Southern View of Leaf Nodes of a PoD, K_TOP=8As Figure 6 shows further, the K_TOP leaf nodes are fully interconnected with the K_LEAF ToP nodes 🫶. That gives you connectivity you can represent as a crossbar when you look at it from the north 🦋. The result: as long as nothing has failed, a packet entering the PoD from the north on any port can be routed to any port in the south of the PoD, and vice versa 💖. That's exactly why it makes sense to call it a "switching matrix" 🔥.
W <---*---> E
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
| | | | | | | | | | | | | | | |
+--------------------------------------------------------+
| o o o o o o o o |
+--------------------------------------------------------+
+--------------------------------------------------------+
| o o o o o o o o |
+--------------------------------------------------------+
+--------------------------------------------------------+
| o o o o o o o o |
+--------------------------------------------------------+
+--------------------------------------------------------+
| o o o o o o o o |
+--------------------------------------------------------+
+--------------------------------------------------------+
| o o o o o o o o |<--+
+--------------------------------------------------------+ |
+--------------------------------------------------------+ |
| o o o o o o o o | |
+--------------------------------------------------------+ |
| | | | | | | | | | | | | | | | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |
^ |
| |
| ---------- ----------------------- |
+----- Leaf Node Top of PoD Node (Spine) --+
---------- -----------------------
Figure 6: Northern View of a PoD's Spines, K_TOP=8Side views of this PoD are shown in Figures 7 and 8 📸
Connecting to ToP Nodes || || || || || || || ||
+----------------------------------------------------------------+ N
| Top-of-PoD Node (Sideways) | ^
+----------------------------------------------------------------+ |
|| || || || || || || || *
+----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
|Leaf| |Leaf| |Leaf| |Leaf| |Leaf| |Leaf| |Leaf| |Leaf| v
|Node| |Node| |Node| |Node| |Node| |Node| |Node| |Node| S
+----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+
|| || || || || || || ||
Connecting to Client Nodes
Figure 7: Side View of a PoD, K_TOP=8, K_LEAF=6 Connecting to ToP Nodes
|| || || || || ||
+----+ +----+ +----+ +----+ +----+ +----+ N
|ToP | |ToP | |ToP | |ToP | |ToP | |ToP | ^
|Node| |Node| |Node| |Node| |Node| |Node| |
+----+ +----+ +----+ +----+ +----+ +----+ *
|| || || || || || |
+------------------------------------------------+ v
| Leaf Node (Sideways) | S
+------------------------------------------------+
Connecting to Client Nodes
Figure 8: Other Side View of a PoD, K_TOP=8, K_LEAF=6, 90-Degree
Turn in E-W Plane from the Previous FigureOkay so, the next step is where it gets really fun 💅✨ Once you've got a PoD, you can squint at the whole thing and treat it as one bigger node 🧚. Its radix is K_POD = K_TOP * K_LEAF 🎀, and the design can recurse, meaning the same trick can be stacked again and again 🌈. But bestie, please make sure the "crossed crossbars" picture is clicking before we go any further 🫶. If it isn't, the next bits will feel like a total maze 😭.
So, the PoDs get wired together through a ToF node at the very top, the north edge of the fabric 🏔️. The ToF is not partitioned if and only if (IIF) every ToP node is connected to every ToF node 💖. This is also called a single-plane configuration, and it's super popular because it's so simple 🍓. There are K_TOP ToF nodes and K_LEAF ToP nodes, because each port of a ToP node connects to a different ToF node 🦋. That means it takes at least P * K_LEAF ports on a ToF node to reach each of the K_LEAF ToP nodes across the P PoDs 🔥. Figure 9 shows this with P=3 PoDs, viewed from above and from two sides 👑. The big view on top is the one from above, with 8 ToF nodes of 3 * 6 ports each, and every ToP node is connected to every ToF node 💅.
[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] <-----+
| | | | | | | | |
[=================================] | --------------
| | | | | | | | +----- ToF
[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] +----- Node ---+
| -------------- |
| v
+-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ <-----+ +-+
| | | | | | | | | | | | | | | | | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] ------------------------- | |
[ |o| |o| |o| |o| |o| |o| |o| |o<--- Physical Port (Ethernet) | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] ------------------------- | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | |
| | | | | | | | | | | | | | | | | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] ---------- | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] <--- ToP Node --------+ | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] (Spine) | | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] ---------- | | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | | |
| | | | | | | | | | | | | | | | -+ +- +-+ v | |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | | --| |--[ ]--| |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | ----- | --| |--[ ]--| |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] +--- PoD ---+ --| |--[ ]--| |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | ----- | --| |--[ ]--| |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | | --| |--[ ]--| |
[ |o| |o| |o| |o| |o| |o| |o| |o| ] | | --| |--[ ]--| |
| | | | | | | | | | | | | | | | -+ +- +-+ | |
+-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+
Figure 9: Fabric Spines and ToFs in Single-Plane Design, 3 PoDsNow, the top view can be squished into a third dimension, where a hidden depth index stands for the PoD number 🧊. That means one PoD can be drawn as a whole class of PoDs, which saves a dimension in the picture 🙈. The ToF node stretches out in depth and vertical space, while the ToP nodes stay constrained to the horizontal dimension 🌸. A port in this 2-D drawing actually represents the whole class of ports sitting in the same spot across all the PoDs that get projected onto its position along the depth axis 💜. Figure 10 shows this off 😍.
/ / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / ]
+-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ ]]
| | | | | | | | | | | | | | | | ] -----------------------
[ |o| |o| |o| |o| |o| |o| |o| |o| ] <-- Top of PoD Node (Spine)
[ |o| |o| |o| |o| |o| |o| |o| |o| ] -----------------------
[ |o| |o| |o| |o| |o| |o| |o| |o| ]]]]
[ |o| |o| |o| |o| |o| |o| |o| |o| ]]] ^^
[ |o| |o| |o| |o| |o| |o| |o| |o| ]] // PoDs
[ |o| |o| |o| |o| |o| |o| |o| |o| ] // (depth)
| |/| |/| |/| |/| |/| |/| |/| |/ //
+-+ +-+ +-+/+-+/+-+ +-+ +-+ +-+ //
^
| --------
+----- ToF Node
--------
Figure 10: Collapsed Northern View of a Fabric for Any Number of PoDsOkay, real talk: a single plane is so simple, but it has a limit 🙃. The radix of the ToF nodes has to be at least P * K_LEAF, and that caps how far you can go ⚠️. Still, a huge perk of a connected, non-partitioned ToF is that every failure can be fixed with simple, non-transitive, positive disaggregation 🎉. That's where nodes advertise more specific prefixes, with a default pointing to the level below them, and that default isn't propagated any further down the fabric 🧚♀️. See Section 6.5.1 for the full scoop 💅. So in other words, non-partitioned ToF nodes can always reach the nodes below them, or withdraw routes to PoDs they can't reach unambiguously 🫶. With all this, positive disaggregation can heal every failure and still let all the ToF nodes know about each other via south reflection 🦋. We'll dig into disaggregation in more detail in Section 6.5 🔥.
To get past the "single-plane limit", the ToF can be partitioned into N identically wired planes 🌈, where N is an integer divider of K_LEAF 💖. The 1:1 ratio and the desired symmetry are still served, just now with (K_TOP*N) ToF nodes, each having (P*K_LEAF/N) ports 👑. N=1 gives you a non-partitioned ToF, which is the superspine 🏛️, and N=K_LEAF gives you a maximally partitioned ToF ✨. Also, if R is any integer divisor of K_LEAF, then N=K_LEAF/R is a feasible number of planes 🍓. R is a redundancy factor, and it counts the number of independent paths between 2 leaves within a plane 💜. It's really convenient for deployments to use a leaf radix that's a power of 2, so they can pick a number of planes that's a lower power of 2 🎀.
Figure 11 splits the ToF into 2 planes with a redundancy factor of R=3 🔥. That means there are 3 non-intersecting paths between any leaf node and any ToF node 😍. In this case, a ToF node has to have at least 3*P ports, and it has to be directly connected to 3 of the 6 ToP nodes (spines) in each PoD 🧚. The ToP nodes are drawn horizontally, with K_TOP=8 ports going north each 🌸.
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
+-| |--| |--| |--| |--| |--| |--| |--| |-+
| | O | | O | | O | | O | | O | | O | | O | | O | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+
+-| |--| |--| |--| |--| |--| |--| |--| |-+
| | O | | O | | O | | O | | O | | O | | O | | O | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+
+-| |--| |--| |--| |--| |--| |--| |--| |-+
| | O | | O | | O | | O | | O | | O | | O | | O | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
Plane 1
----------- . ------------ . ------------ . ------------ . --------
Plane 2
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
+-| |--| |--| |--| |--| |--| |--| |--| |-+
| | O | | O | | O | | O | | O | | O | | O | | O | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+
+-| |--| |--| |--| |--| |--| |--| |--| |-+
| | O | | O | | O | | O | | O | | O | | O | | O | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+
+-| |--| |--| |--| |--| |--| |--| |--| |-+
| | O | | O | | O | | O | | O | | O | | O | | O | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
^
|
| ---------------------
+----- ToF Node Across Depth
---------------------
Figure 11: Northern View of a Multi-Plane ToF Level, K_LEAF=6, N=2At the extreme end of the spectrum 🌌, it's even possible to fully partition the ToF with N=K_LEAF and R=1, while still keeping connectivity between each leaf node and each ToF node 💅. In that case, the ToF node connects to a single port per PoD, so it looks like a single port in the projected view shown in Figure 12 ✨. The number of ports needed on the ToF node is more than or equal to P, i.e., the number of PoDs 🩷.
Plane 1
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ -+
+-| |--| |--| |--| |--| |--| |--| |--| |-+ |
| | O | | O | | O | | O | | O | | O | | O | | O | | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |
----------- . ------------------- . ------------ . ------- |
Plane 2 |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ |
| | O | | O | | O | | O | | O | | O | | O | | O | | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |
----------- . ------------ . ---- . ------------ . ------- |
Plane 3 |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ |
| | O | | O | | O | | O | | O | | O | | O | | O | | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |
----------- . ------------ . ------------------- . --------+<-+
Plane 4 | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ | |
| | O | | O | | O | | O | | O | | O | | O | | O | | | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | |
----------- . ------------ . ------------ . ---- . ------- | |
Plane 5 | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ | |
| | O | | O | | O | | O | | O | | O | | O | | O | | | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | |
----------- . ------------ . ------------ . -------------- | |
Plane 6 | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ | |
| | O | | O | | O | | O | | O | | O | | O | | O | | | |
+-| |--| |--| |--| |--| |--| |--| |--| |-+ | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ -+ |
^ |
| |
| ---------------- ------------- |
+----- ToF Node Class of PoDs ---+
---------------- -------------Figure 12: Northern View of a Maximally Partitioned ToF Level, R=1
5.3. Fallen Leaf Problem
Okay so, like I said earlier, RIFT is lowkey anisotropic 🧭, which just means it's built for fabrics with a north-south orientation and a high level of interleaving paths 🕸️. In a non-partitioned fabric, a total loss of connectivity between a ToF node at the north and a leaf node at the south is a very rare but possible moment 😳, and it's fully healed by positive disaggregation as described in Section 6.5.1 ✨. But in large fabrics, or fabrics built from switches with a low radix, the ToF may often get partitioned into planes 💔. That makes it more likely that a given leaf is only reachable from a subset of the ToF nodes, so we need some further considerations 💅.
A "fallen leaf" 🍂 is a leaf that can be reached by only a subset of ToF nodes due to missing connectivity. If R is the redundancy factor, then it takes at least R breakages to reach a "fallen leaf" situation.
In a maximally partitioned fabric, the redundancy factor is R=1, so any breakage in the fabric will cause one or more fallen leaves in the affected plane. R=2 guarantees that a single breakage will not cause a fallen leaf. However, not all cases require disaggregation 🙌. The following cases do not require particular action:
- If a southern link on a node goes down, then connectivity through that node is lost for all nodes south of that link. There is no need to disaggregate since the connectivity to this node is lost for all spine nodes in the same fashion.
- If a ToF node goes down, then northern traffic towards it is routed via alternate ToF nodes in the same plane and there is no need to disaggregate routes.
In a general manner, the mechanism of non-transitive, positive disaggregation is sufficient when the disaggregating ToF nodes collectively connect to all the ToP nodes in the broken plane. This happens in the following case:
- If the breakage is the last northern link from a ToP node to a ToF node going down, then the fallen leaf problem affects only that ToF node, and the connectivity to all the nodes in the PoD is lost from that ToF node. This can be observed by other ToF nodes within the plane where the ToP node is located and positively disaggregated within that plane.
On the other hand, there is a need to disaggregate the routes to Fallen Leaves within the plane in a transitive fashion, that is, all the way to the other leaves, in the following cases:
- If the breakage is the last northern link from a leaf node within a plane (there is only one such link in a maximally partitioned fabric) that goes down, then connectivity to all unicast prefixes attached to the leaf node is lost within the plane where the link is located. Southern Reflection by a leaf node, e.g., between ToP nodes, if the PoD has only 2 levels, happens in between planes, allowing the ToP nodes to detect the problem within the PoD where it occurs and positively disaggregate. The breakage can be observed by the ToF nodes in the same plane through the north flooding of TIEs from the ToP nodes. However, the ToF nodes need to be aware of all the affected prefixes for the negative, possibly transitive, disaggregation to be fully effective (i.e., a node advertising in the control plane that it cannot reach a certain more specific prefix than the default prefix, whereas such disaggregation in the extreme condition must be propagated further down southbound). The problem can also be observed by the ToF nodes in the other planes through the flooding of North TIEs from the affected leaf nodes, together with non-node North TIEs, which indicate the affected prefixes. To be effective in that case, the positive disaggregation must reach down to the nodes that make the plane selection, which are typically the ingress leaf nodes. The information is not useful for routing in the intermediate levels.
- If the breakage is a ToP node in a maximally partitioned fabric (in which case it is the only ToP node serving the plane in that PoD that goes down), then the connectivity to all the nodes in the PoD is lost within the plane where the ToP node is located. Consequently, all leaves of the PoD fall in this plane. Since the Southern Reflection between the ToF nodes happens only within a plane, ToF nodes in other planes cannot discover fallen leaves in a different plane. They also cannot determine beyond their local plane whether a leaf node that was initially reachable has become unreachable. As the breakage can be observed by the ToF nodes in the plane where the breakage happened, the ToF nodes in the plane need to be aware of all the affected prefixes for the negative disaggregation to be fully effective. The problem can also be observed by the ToF nodes in the other planes through the flooding of North TIEs from the affected leaf nodes if the failing ToP node is directly connected to its leaf nodes, which can detect the link going down. Then again, the knowledge of the failure at the ToF level can only be useful if it is propagated transitively to all the leaves; it is useless above that level since the decision of placing a packet in a plane happens at the leaf that injects the packet in the fabric.
These abstractions are rolled back into a simplified example that shows that in Figure 3 the loss of the link between spine node 3 and leaf node 3 will make leaf node 3 a fallen leaf for ToF nodes in plane C. Worse, if the cabling was never present in the first place, plane C will not even be able to know that such a fallen leaf exists. Hence, partitioning without further treatment results in two grave problems 😱:
- Leaf node 1 trying to route to leaf node 3 must not choose spine node 3 in plane C as its next hop since it will inevitably drop the packet when forwarding using default routes or do excessive bow-tying. This information must be in its routing table.
- A path computation trying to deal with the problem by distributing host routes may only form paths through leaves. The flooding of information about leaf node 3 would have to go up to ToF nodes in planes A, B, and D and then "loopback" over other leaves to ToF C, leading in extreme cases to traffic for leaf node 3 when presented to plane C taking an "inverted fabric" path where leaves start to serve as ToFs, at least for the duration of a protocol's convergence. 🙃
5.4. Discovering Fallen Leaves
When aggregation is used, RIFT deals with fallen leaves by ensuring that all the ToF nodes share the same north topology database. This happens naturally in single-plane design by the means of northbound flooding and south reflection but needs additional considerations in multi-plane fabrics. To enable routing to fallen leaves in multi-plane designs, RIFT requires additional interconnection across planes between the ToF nodes, e.g., using rings as illustrated in Figure 13. Other solutions are possible, but they either need more cabling or end up having much longer flooding paths and/or single points of failure. 🔗
In detail, by reserving at least two ports on each ToF node, it is possible to connect them together by interplane bidirectional rings as illustrated in Figure 13. The rings will be used to exchange full north topology information between planes. All ToFs having the same north topology allows, by the means of transitive, negative disaggregation described in Section 6.5.2, to efficiently fix any possible fallen leaf scenario. Somewhat as a side effect, the exchange of information fulfills the requirement for a full view of the fabric topology at the ToF level without the need to collate it from multiple points. 💍
(Artwork only available as SVG: see
https://www.rfc-editor.org/rfc/rfc9692.html)Figure 13: Using Rings to Bring All Planes and Bind Them at the ToF
5.5. Addressing the Fallen Leaves Problem ✨
Okay so the "fallen leaf" problem has a sneaky side effect 😩: some prefixes hanging off the fallen leaf become unreachable from some of the ToF nodes. RIFT has two fixes for this, called positive disaggregation and negative disaggregation 🔧. Both of them flood the matching kind of South TIEs to advertise the impacted prefix(es) 📣.
Here's the vibe check. When a positive South TIE is used for disaggregation, it still means what it always means: "I can reach this prefix of a given length, and everything inside it." A negative route advertisement is the opposite 🙅♀️. It says the origin cannot route to the advertised prefix.
Positive disaggregation starts with a router that can still reach the advertised prefix, and it is not transitive 🚫. The receiver does not make its own TIEs or flood them south just because a higher-level node sent it positive disaggregation advertisements. The result is that traffic to the impacted prefix follows the longest match, and it's limited to the northbound routers that advertised the more specific route 🎯.
Negative disaggregation, on the other hand, can be transitive 🔁. It gets propagated south once all the possible routes have been advertised as negative exceptions. A negative route advertisement only does anything when the negative prefix is aggregated by a positive route advertisement for a shorter prefix 🧩. In that case, the negative advertisement "punches out a hole" in the positive route in the routing table 🕳️. The positive prefix stays reachable through the originator, except that the negative prefix removes certain next-hop neighbors. The exact procedures are explained in detail in Section 6.5.2.3.
When the ToF switches are not partitioned into multiple planes, the southbound flooding of positive disaggregation from the ToF nodes that can still reach the impacted prefix is usually enough to cover all the switches at the next level south, typically the ToP nodes 🌸. If all of those switches know about the disaggregation, they basically form a ceiling that catches all the northbound traffic and forwards it to the ToF nodes that advertised the more specific route 🙌. So in that setup, positive disaggregation on its own is enough to solve the fallen leaf problem 💖.
Now, when the fabric is partitioned into planes, things get messier 🥲. Positive disaggregation from ToF nodes in other planes doesn't reach the ToP switches in the affected plane, so it can't fix the fallen leaves problem there. In other words, a breakage in one plane can only be solved inside that plane. Also, the plane a packet ends up in is usually picked at the leaf level, so the disaggregation has to be transitive and reach all the leaves 🍃. That's when negative disaggregation becomes necessary 🫶. The details on how RIFT deals with fallen leaves in an optimal way are in Section 6.5.2.
6. Specification 👑
This section lays out the protocol in a normative way, using either prescriptive procedures or behavior defined by Finite State Machines (FSMs) 🧠.
The FSMs work the way you'd expect 💅. Each one is presented as the states a neighbor can be in, the events that can happen, and the actions that run when it transitions between states while processing an event 🔄.
Actions get performed before the end state is assumed ✅.
An FSM can queue events against itself to chain actions, or queue them against other FSMs in the specification 📬. Events are always processed in the order they were queued, no skipping the line 🙅♀️.
Because of that, "On Entry" actions for an FSM state run every single time, right before the state is entered, which means after any transition from the previous state 🚪. "On Exit" actions also run every single time, and they run immediately when a state is exited, which means before any transitions toward the target state happen 🏃♀️.
Here's the important part 🚨. Any attempt to move from one state to another when an event arrives and no action is specified MUST be treated as an unrecoverable error, and the protocol MUST reset all adjacencies and discard all the states. In plain English, that means forcing the FSM back to _OneWay_ and flushing all of the queues that hold flooding information 💥.
The data structures and FSMs in this document are conceptual 🧁. They don't have to be implemented exactly the way they're described here. An implementation counts as conforming as long as it supports the described functionality and shows externally observable behavior that's equivalent to the standardized FSMs.
The FSMs can use "timers" for different situations ⏱️. Timers get started through actions, and when one expires, it queues the matching event to be processed.
The word "holdtime" shows up a lot as shorthand for "holddown timer" 🫧. It refers either to the length of the holding-down period or to the timer that expires after that period. These timers "holddown" the state inside an FSM, and that state gets cleaned up if the machine triggers a _HoldtimeExpired_ event 🧹.
6.1. Transport 🚚
All normative RIFT packet structures and their contents are defined in the Thrift [thrift] models in Section 7 📦. The packet structure itself is defined in _ProtocolPacket_, which holds the packet header in _PacketHeader_ and the packet contents in _PacketContent_. _PacketContent_ is a union of the LIE, TIE, TIDE, and TIRE packets, which are defined in _LIEPacket_, _TIEPacket_, _TIDEPacket_, and _TIREPacket_, respectively 🧩.
In terms of bits on the wire, it's the _ProtocolPacket_ that gets serialized and carried in an envelope defined in Section 6.9.3 🎁. That envelope sits inside a UDP frame. It provides security and lets you validate or modify several important fields without doing a Thrift deserialization, for performance and security reasons 🔐. Security models and procedures are explained further in Section 9 🛡️.
6.2. Link (Neighbor) Discovery (LIE Exchange) 🤝
RIFT LIE exchange auto-discovers neighbors, negotiates RIFT ZTP parameters, and discovers miscablings 🕵️♀️. Under normal conditions, the formation goes from _OneWay_ to _TwoWay_ and then to the _ThreeWay_ state 📈. At that point, it's ready to exchange TIEs, as described in Section 6.3 🚀. The adjacency exchanges RIFT ZTP information (Section 6.7) in any of the states, so you don't have to reach _ThreeWay_ for ZTP to work 🙌.
RIFT supports any combination of IPv4 and IPv6 addressing on the fabric to form adjacencies, including link-local scope 🌈. It also has an extra capability for forwarding paths that can carry IPv4 packets even when only IPv6 addressing is present 🦋.
IPv4 LIE exchange happens by default over a well-known IPv4 multicast address [RFC2365]. That address can also be configured administratively, for example with a local scope 🎀. For IPv6 [RFC8200], the exchange happens over the link-local multicast scope [RFC4291] address, which is either configured or well-known 💜. In both cases, a destination UDP port defined in the schema (Section 7.2) is used, unless it's configured differently ⚙️.
LIEs MUST be sent with an IPv4 Time to Live (TTL) or an IPv6 Hop Limit (HL) of either 1 or 255. That keeps RIFT information from reaching beyond a single Layer 3 (L3) next hop in the topology 🚧. Notice that for the allocated link-local scope IP multicast address, a TTL of 1 is the more logical choice 🧠. A TTL of 255 might, in some environments, lead to an early drop because it looks suspicious for a packet addressed to that destination 🙈. LIEs SHOULD be sent with network control precedence, unless an implementation is prevented from doing so [RFC2474] 👑.
Any LIE packet received on an address that is neither the well-known nor the configured multicast address, nor a broadcast address, MUST be discarded 🗑️.
The originating port of the LIE has no further significance beyond identifying where it came from 📍. LIEs are exchanged over all links running RIFT 🔗.
An implementation may listen and send LIEs on IPv4 and/or IPv6 multicast addresses 🎧. A node MUST NOT originate LIEs on an address family if it doesn't process received LIEs on that family 🙅♀️. LIEs on the same link count as part of the same LIE FSM, no matter which address family they arrive on 🫂. In unnumbered or link-local address cases, the LIE source address may not identify the peer uniquely. So the response transmission MUST happen over the same interface the LIEs were received on 🔁. A node may use any of the adjacency's source addresses it saw in LIEs on that specific interface during adjacency formation to send TIEs (Section 6.3.3) 📨. Which means an implementation MUST be ready to accept TIEs on all the addresses it used as sources of LIE frames 💌.
A simplified version MAY be implemented on platforms with limited multicast support, like Internet of Things (IoT) devices 🤖. It does this by sending and receiving LIE frames on IPv4 subnet broadcast addresses or IPv6 all-routers multicast addresses 📡. However, this technique is less optimal and presents a wider attack surface from a security perspective, so it should hence be used only as a last resort 🚨. Honestly, this is one to avoid unless you really have no other choice 😬.
A _ThreeWay_ adjacency (as defined in the glossary) over any address family implies support for IPv4 forwarding if the _ipv4_forwarding_capable_ flag in _LinkCapabilities_ is set to true 🥳. If there are no IPv4 LIEs with _ipv4_forwarding_capable_ set to true, a node MUST forward IPv4 packets using gateways discovered on IPv6-only links that advertise this capability 🧭. How to discover the corresponding IPv6 gateway is out of scope for this specification, and it may be implementation-specific 🤷♀️. The fabric is expected to support the same type of address family forwarding on all of its links. Any other combination is outside the scope of this specification 🙃. If IPv4 forwarding is supported on an interface, _ipv4_forwarding_capable_ MUST be set to true for all LIEs advertised from that interface ✅. If IPv4 and IPv6 LIEs indicate contradicting information, the protocol behavior is unspecified 🌀. A node sending IPv4 LIEs MUST set the _ipv4_forwarding_capable_ flag to true on all LIEs advertised from that interface 💯.
Operating a fabric where only some of the links support forwarding on an address family, or have an address in a family while others don't, is outside the scope of this specification 🚫.
Any attempt to build IPv6 forwarding over IPv4-only adjacencies is also outside the scope of this specification 🙅♀️.
Table 1 outlines the protocol behavior for LIE exchange across different address family combinations 📋. Table 2 outlines how neighbors forward traffic based on the _ipv4_forwarding_capable_ flag setting across those same combinations 🗂️. The table is symmetric, meaning you can swap the local and remote columns to build the remaining combinations 🔄.
The specific forwarding implementation that supports the described behavior is out of scope for this document 🫶.
okay bestie, let's look at which address family combos are giving 💅
+==========+==========+==========================================+
| Local | Remote | LIE Exchange Behavior |
| Neighbor | Neighbor | |
| Address | Address | |
| Family | Family | |
+==========+==========+==========================================+
| IPv4 | IPv4 | LIEs and TIEs are exchanged over IPv4 |
| | | only. The local neighbor receives TIEs |
| | | from remote neighbors on any of the LIE |
| | | source addresses. |
+----------+----------+------------------------------------------+
| IPv6 | IPv6 | LIEs and TIEs are exchanged over IPv6 |
| | | only. The local neighbor receives TIEs |
| | | from remote neighbors on any of the LIE |
| | | source addresses. |
+----------+----------+------------------------------------------+
| IPv4, | IPv6 | The local neighbor sends LIEs for both |
| IPv6 | | IPv4 and IPv6, while the remote neighbor |
| | | only sends LIEs for IPv6. The resulting |
| | | adjacency will exchange TIEs over IPv6 |
| | | on any of the IPv6 LIE source addresses. |
+----------+----------+------------------------------------------+
| IPv4, | IPv4, | LIEs and TIEs are exchanged over IPv6 |
| IPv6 | IPv6 | and IPv4. TIEs are received on any of |
| | | the IPv4 or IPv6 LIE source addresses. |
| | | The local neighbor receives TIEs from |
| | | the remote neighbors on any of the IPv4 |
| | | or IPv6 LIE source addresses. |
+----------+----------+------------------------------------------+
| IPv4, | IPv4 | The local neighbor sends LIEs for both |
| IPv6 | | IPv4 and IPv6, while the remote neighbor |
| | | only sends LIEs for IPv4. The resulting |
| | | adjacency will exchange TIEs over IPv4 |
| | | on any of the IPv4 LIE source addresses. |
+----------+----------+------------------------------------------+
Table 1: Control Plane Behavior for Neighbor Address Family
CombinationsNow the forwarding side 🦋
+==========+==========+==========================================+
| Local | Remote | Forwarding Behavior |
| Neighbor | Neighbor | |
| Address | Address | |
| Family | Family | |
+==========+==========+==========================================+
| IPv4 | IPv4 | Only IPv4 traffic can be forwarded. |
+----------+----------+------------------------------------------+
| IPv6 | IPv6 | If either neighbor sets |
| | | _ipv4_forwarding_capable_ to false, only |
| | | IPv6 traffic can be forwarded. If both |
| | | neighbors set _ipv4_forwarding_capable_ |
| | | to true, IPv4 traffic is also forwarded |
| | | via IPv6 gateways. |
+----------+----------+------------------------------------------+
| IPv4, | IPv6 | If the remote neighbor sets |
| IPv6 | | _ipv4_forwarding_capable_ to false, only |
| | | IPv6 traffic can be forwarded. If both |
| | | neighbors set _ipv4_forwarding_capable_ |
| | | to true, IPv4 traffic is also forwarded |
| | | via IPv6 gateways. |
+----------+----------+------------------------------------------+
| IPv4, | IPv4, | IPv4 and IPv6 traffic can be forwarded. |
| IPv6 | IPv6 | If IPv4 and IPv6 LIEs advertise |
| | | conflicting _ipv4_forwarding_capable_ |
| | | flags, the behavior is unspecified. |
+----------+----------+------------------------------------------+
| IPv4, | IPv4 | IPv4 traffic can be forwarded. |
| IPv6 | | |
+----------+----------+------------------------------------------+
Table 2: Forwarding Behavior for Neighbor Address Family
CombinationsBestie, RIFT is NOT giving "switch off one address family mid-adjacency" 🙅♀️. Once an adjacency is formed you can't selectively disable address families, you can't turn off IPv4 forwarding capability, and you can't change local addresses while you're in ThreeWay state. So if your link has hit ThreeWay with a neighbor over IPv4 and/or IPv6 and you want to drop a family, change a local address, or stop IPv4 forwarding, you MUST tear the adjacency down and rebuild it. You MUST also wipe any state you stored about the remote side, like the LIE source addresses tied to it 🧹
Unless RIFT ZTP is used as described in Section 6.7, every node is provisioned with the level it's operating at and advertises it in the level of the PacketHeader schema element. It MAY also be provisioned with its PoD. If the level isn't provisioned, it's simply not present in the optional PacketHeader schema element and gets established by ZTP procedures, if feasible. If PoD isn't provisioned, it's governed by the LIEPacket schema element, assuming the common.default_pod value. So yes, switches except ToF don't need to be configured at all 🧚 All the info needed to configure everything is exchanged in the LIEPacket and PacketHeader, or the node derives it automatically.
Leaf flags get their full definitions in Section 6.7, since they affect level and adjacency formation. They're carried in HierarchyIndications.
A node MUST form a ThreeWay adjacency if, at a minimum, the following first order logic conditions are satisfied on a LIE packet, as specified by the LIEPacket schema element and received on a link. Such a LIE counts as a "minimally valid" LIE. Observe that, depending on the FSM involved and its state, further conditions may be checked, and even a minimally valid LIE can end up ultimately invalid if any of the additional conditions fail:
- the neighboring node is running the same major schema version as indicated in the
major_versionelement inPacketHeaderand - the neighboring node uses a valid System ID (i.e., a value different from
IllegalSystemID) in thesenderelement inPacketHeaderand - the neighboring node uses a different System ID than the node itself and
- (the advertised MTU values in the
LiePacketelement match on both sides, while a missing MTU in theLiePacketelement is interpreted asdefault_mtu_size) and - both nodes advertise defined level values in the
levelelement inPacketHeaderand - [ a. the node is at the
leaf_levelvalue and does not already have anyThreeWayadjacencies to nodes that are at the Highest AdjacencyThreeWay(HAT), as defined in Section 6.7.1, with a level that is different than the adjacent node or b. the node is not at theleaf_levelvalue and the neighboring node is at theleaf_levelvalue or c. both nodes are at theleaf_levelvalue and both indicate support for that described in Section 6.8.9 or d. neither node is at theleaf_levelvalue and the neighboring node is, at most, one level away. ]
LIEs arriving with an IPv4 Time to Live (TTL) or an IPv6 Hop Limit (HL) different than 1 or 255 MUST be ignored. 🙈
6.2.1. LIE Finite State Machine ✨
This section specifies the precise, normative LIE FSM, which is also shown in Figure 14. Some sets of actions show up over and over, so they're summarized into well-known procedures, which is very neat of them 💖
Events are generated pretty fine grained, especially when they're flagging problems with adjacency-forming conditions. That makes it way easier to track down issues once this is out in the wild 🔍
The initial state is OneWay.
The machine sends LIEs proactively on several transitions so adjacency bring-up is faster, without waiting for the corresponding timer tic. Speedy 🏎️💨
Enter
|
V
+-----------+
| OneWay |<----+
| | | TimerTick
| | | LevelChanged
| | | HALChanged
| | | HATChanged
| | | HALSChanged
| | | LieRcvd
| | | NeighborDroppedReflection
| | | NeighborChangedLevel
| | | NeighborChangedAddress
| | | UnacceptableHeader
| | | MTUMismatch
| | | NeighborChangedMinorFields
| | | HoldtimeExpired
| | | FloodLeadersChanged
| | | SendLie
| | | UpdateZTPOffer
| |-----+
| |
| |<--------------------- (ThreeWay)
| |--------------------->
| | ValidReflection
| |
| |---------------------> (Multiple
| | MultipleNeighbors Neighbors
+-----------+ Wait)
^ |
| |
| | NewNeighbor
| V
(TwoWay) (OneWay)
| ^
| | NeighborChangedLevel
| | NeighborChangedAddress
| | UnacceptableHeader
| | MTUMismatch
| | HoldtimeExpired
| |
V |
+-----------+
| TwoWay |<----+
| | | TimerTick
| | | LevelChanged
| | | HALChanged
| | | HATChanged
| | | HALSChanged
| | | LieRcvd
| | | FloodLeadersChanged
| | | SendLie
| | | UpdateZTPOffer
| |-----+
| |
| |<----------------------
| |----------------------> (Multiple
| | NewNeighbor Neighbors
| | Wait)
| | MultipleNeighbors
+-----------+
^ |
| | ValidReflection
| V
(ThreeWay)
(TwoWay) (OneWay)
^ | ^
| | | LevelChanged
| | | NeighborChangedLevel
| | | NeighborChangedAddress
| | | UnacceptableHeader
| | | MTUMismatch
| | | HoldtimeExpired
NeighborDropped- | | |
Reflection | | |
| V |
+-----------+ |
| ThreeWay |-----+
| |
| |<----+
| | | TimerTick
| | | HALChanged
| | | HATChanged
| | | HALSChanged
| | | LieRcvd
| | | ValidReflection
| | | FloodLeadersChanged
| | | SendLie
| | | UpdateZTPOffer
| |-----+
| |----------------------> (Multiple
| | MultipleNeighbors Neighbors
+-----------+ Wait)
(TwoWay) (ThreeWay)
| |
V V
+------------+
| Multiple |<----+
| Neighbors | | TimerTick
| Wait | | HALChanged
| | | HATChanged
| | | HALSChanged
| | | LieRcvd
| | | ValidReflection
| | | NeighborDroppedReflection
| | | NeighborChangedBFDCapability
| | | NeighborChangedAddress
| | | UnacceptableHeader
| | | MTUMismatch
| | | HoldtimeExpired
| | | MultipleNeighbors
| | | FloodLeadersChanged
| | | SendLie
| | | UpdateZTPOffer
| |-----+
| |
| |<---------------------------
| |---------------------------> (OneWay)
| | LevelChanged
+------------+ MultipleNeighborsDone
Figure 14: LIE FSMOkay so the FSM has a few well-known procedures 💅 that it keeps using, and they're basically the building blocks of the whole state machine. Let's go bestie 🎀
- PUSH Event 📮: queues an event up so the FSM runs it once this action is done (on exit) ✨
- CLEANUP 🧹: the FSM conceptually keeps a "current neighbor" variable 🧠 with whatever came in the remote node's LIE after it's been checked against the LIE validation rules. If that LIE turns out to be invalid, the existing state held by a "current neighbor" MUST be deleted 🗑️💀
- SEND_LIE 📣: makes and sends a fresh LIE packet, and while it's doing that it also:
- reflects the
neighborelement the way ValidReflection describes 🪞, - sets the necessary
not_a_ztp_offervariable if the level came from the last-known neighbor on this interface, and - sets the
you_are_flood_repeatervariable to the computed value 🔥.
- reflects the
- PROCESS_LIE 🔍: this is the big bouncer 🚪 that checks an incoming LIE in order:
- if the LIE has a major version that isn't equal to this node's major version or its System ID is equal to this node's System ID or
IllegalSystemID, then CLEANUP, else - if both sides advertise Layer 2 MTU values and the MTU in the received LIE doesn't match the MTU the local system advertises or at least one of the nodes doesn't advertise an MTU value and the advertising node's LIE doesn't match the
default_mtu_sizeof the system that didn't advertise one, then CLEANUP, then PUSH UpdateZTPOffer, then PUSH MTUMismatch, else - if the LIE has an undefined level or this node's level is undefined or this node is a leaf and the remote level is lower than HAT or the LIE's level isn't leaf and its difference is more than one from this node's level, then CLEANUP, then PUSH UpdateZTPOffer, then PUSH UnacceptableHeader, else
- PUSH UpdateZTPOffer, then build a temporary new neighbor structure with values from the LIE. If no current neighbor exists, set the current neighbor to the new neighbor, PUSH NewNeighbor event, and CHECK_THREE_WAY, else:
- a. if the current neighbor's System ID differs from the LIE's System ID, then PUSH MultipleNeighbors, else
- b. if the current neighbor's stored level differs from the LIE's level, then PUSH NeighborChangedLevel, else
- c. if the current neighbor's stored IPv4/v6 address differs from the LIE's address, then PUSH NeighborChangedAddress, else
- d. if any of the neighbor's flood address port, name, or local LinkID changed, then PUSH NeighborChangedMinorFields
- e. CHECK_THREE_WAY
- if the LIE has a major version that isn't equal to this node's major version or its System ID is equal to this node's System ID or
- CHECK_THREE_WAY 🎯: if the current state is OneWay, do nothing, else:
- if the LIE packet doesn't contain a neighbor, then if the current state is ThreeWay, PUSH NeighborDroppedReflection, else
- if the packet reflects this System ID and local port and the state is ThreeWay, then PUSH the ValidReflection event, else PUSH the MultipleNeighbors event.
Now the states 👑. The FSM can be in one of four vibes:
- OneWay 🌱: the starting point 🏁. The router hasn't received any valid LIEs from a neighbor yet, so it's a little lonely in here 🥺
- TwoWay 🤝: a node lands here once it's gotten a minimally valid LIE from a neighbor, but not a ThreeWay valid one yet.
- ThreeWay 🔥: this one means ThreeWay valid LIEs from a neighbor have been received 🙌. Once you hit this state, the link can get advertised in the
neighborselement inNodeTIEElement✨ - MultipleNeighborsWait ⏳: this happens when more than two nodes suddenly realize they're on the same link, or a remote node got reconfigured or rebooted super fast without going back to OneWay first 🙈. Each time this event happens, it SHOULD generate a notification to help operational deployments 📢
Next up are the events 📬, aka the things that can happen to the FSM:
- TimerTick ⏰: a one-second tick. An implementation-specific mechanism that's outside the scope of this spec hands it to the FSM once a second 🫶. It gets quietly ignored if there's no transition for it.
- LevelChanged 🪜: the node's level got changed by ZTP or by configuration. This one comes from the ZTP FSM.
- HALChanged 🏆: the best HAL that ZTP computed has changed. Comes from the ZTP FSM.
- HATChanged 🎚️: the HAT that ZTP computed has changed. Also comes from the ZTP FSM.
- HALSChanged 🧾: the set of HAL-offering systems that ZTP computed has changed. Yep, ZTP FSM again.
- LieRcvd 📩: a LIE was received on the interface.
- NewNeighbor 👋: a new neighbor shows up in the received LIE.
- ValidReflection 🪞: this node got a valid reflection back from the neighbor. In other words, every element in the
neighborelement inLiePackethas values that match this link 💯. - NeighborDroppedReflection 👻: the node lost a reflection it was holding from the neighbor. That means the
neighborelement inLiePacketdoesn't match this node or just isn't there at all 😭. - NeighborChangedLevel 🪜🔄: the neighbor changed the level it's advertising compared to the one held before.
- NeighborChangedAddress 🏠: the neighbor changed its IP address, meaning the LIE came in from an address that's different from the previous LIEs. Those changes will mess with the sockets used to listen to TIEs, TIREs, and TIDEs 🧦.
- UnacceptableHeader 🚫: an unacceptable header was received.
- MTUMismatch 📏: the MTU doesn't match up 🙃.
- NeighborChangedMinorFields 🧩: some minor fields in the neighbor's LIE changed.
- HoldtimeExpired ⌛: the adjacency holddown timer ran out.
- MultipleNeighbors 👯: there's more than one neighbor on the interface.
- MultipleNeighborsDone ✅: the multiple neighbors timer expired.
- FloodLeadersChanged 🦸: the node's election algorithm decided on a new set of flood leaders.
- SendLie 📤: send a LIE out.
- UpdateZTPOffer 🎁: update this node's ZTP offer. This one gets sent to the ZTP FSM.
And finally the actions 🎬, which are what the FSM does when a given event hits a given state. Every single one of these is part of the plan ✨
- on HATChanged in OneWay, finishes in OneWay: store HAT 💾
- on FloodLeadersChanged in OneWay, finishes in OneWay: update the
you_are_flood_repeaterLIE elements based on the flood leader election results 🗳️ - on UnacceptableHeader in OneWay, finishes in OneWay: no action 🙅
- on NeighborChangedMinorFields in OneWay, finishes in OneWay: no action 🙅
- on SendLie in OneWay, finishes in OneWay: SEND_LIE 📣
- on HALSChanged in OneWay, finishes in OneWay: store the HALS 🧾
- on MultipleNeighbors in OneWay, finishes in MultipleNeighborsWait: start the multiple neighbors timer with the interval
multiple_neighbors_lie_holdtime_multiplier*default_lie_holdtime⏳✨ - on NeighborChangedLevel in OneWay, finishes in OneWay: no action 🙅
- on LieRcvd in OneWay, finishes in OneWay: PROCESS_LIE 🔍
- on MTUMismatch in OneWay, finishes in OneWay: no action 🙅
- on ValidReflection in OneWay, finishes in ThreeWay: no action 🪞🔥
- on LevelChanged in OneWay, finishes in OneWay: update the level with the event value, then PUSH the SendLie event 🪜📤
- on HALChanged in OneWay, finishes in OneWay: store the new HAL 💾
- on HoldtimeExpired in OneWay, finishes in OneWay: no action 🙅
- on NeighborChangedAddress in OneWay, finishes in OneWay: no action 🙅
- on NewNeighbor in OneWay, finishes in TwoWay: PUSH the SendLie event 👋📤
- on UpdateZTPOffer in OneWay, finishes in OneWay: send the offer to the ZTP FSM 🎁💅
- on
NeighborDroppedReflectionin OneWay finishes in OneWay: no action 🫠 - on
TimerTickin OneWay finishes in OneWay: PUSHSendLieevent 📣 - on
FloodLeadersChangedin TwoWay finishes in TwoWay: update you_are_flood_repeater LIE elements based on the flood leader election results 👑 - on
UpdateZTPOfferin TwoWay finishes in TwoWay: send the offer to the ZTP FSM 🎁 - on
NewNeighborin TwoWay finishes in MultipleNeighborsWait: PUSH theSendLieevent 💅 - on
ValidReflectionin TwoWay finishes in ThreeWay: no action ✨ - on
LieRcvdin TwoWay finishes in TwoWay:PROCESS_LIE🔍 - on
UnacceptableHeaderin TwoWay finishes in OneWay: no action 🙈 - on
HALChangedin TwoWay finishes in TwoWay: store the new HAL 💾 - on
HoldtimeExpiredin TwoWay finishes in OneWay: no action 💀 - on
LevelChangedin TwoWay finishes in TwoWay: update the level with the event value 📈 - on
TimerTickin TwoWay finishes in TwoWay: PUSHSendLieevent, if last valid LIE was received more than holdtime ago as advertised by the neighbor, then PUSH theHoldtimeExpiredevent ⏰ - on
HATChangedin TwoWay finishes in TwoWay: store HAT 🗂️ - on
NeighborChangedLevelin TwoWay finishes in OneWay: no action 🫶 - on
HALSChangedin TwoWay finishes in TwoWay: store the HALS 🌸 - on
MTUMismatchin TwoWay finishes in OneWay: no action 📏 - on
NeighborChangedAddressin TwoWay finishes in OneWay: no action 🏠 - on
SendLiein TwoWay finishes in TwoWay:SEND_LIE📤 - on
MultipleNeighborsin TwoWay finishes in MultipleNeighborsWait: start multiple neighbors timer with the intervalmultiple_neighbors_lie_holdtime_multiplier * default_lie_holdtime⏳ - on
TimerTickin ThreeWay finishes in ThreeWay: PUSH theSendLieevent, if the last valid LIE was received more than holdtime ago as advertised by the neighbor, then PUSH theHoldtimeExpiredevent 🫧 - on
LevelChangedin ThreeWay finishes in OneWay: update the level with the event value 🌈 - on
HATChangedin ThreeWay finishes in ThreeWay: store HAT 🧚 - on
MTUMismatchin ThreeWay finishes in OneWay: no action 🍓 - on
UnacceptableHeaderin ThreeWay finishes in OneWay: no action 🙃 - on
MultipleNeighborsin ThreeWay finishes in MultipleNeighborsWait: start multiple neighbors timer with the intervalmultiple_neighbors_lie_holdtime_multiplier * default_lie_holdtime⏳ - on
NeighborChangedLevelin ThreeWay finishes in OneWay: no action 🌷 - on
HALSChangedin ThreeWay finishes in ThreeWay: store the HALS 💜 - on
LieRcvdin ThreeWay finishes in ThreeWay:PROCESS_LIE🕵️♀️ - on
FloodLeadersChangedin ThreeWay finishes in ThreeWay: update you_are_flood_repeater LIE elements based on the flood leader election results, PUSH theSendLieevent 👑 - on
NeighborDroppedReflectionin ThreeWay finishes in TwoWay: no action 🫠 - on
HoldtimeExpiredin ThreeWay finishes in OneWay: no action 💅 - on
ValidReflectionin ThreeWay finishes in ThreeWay: no action 🦋 - on
UpdateZTPOfferin ThreeWay finishes in ThreeWay: send the offer to the ZTP FSM 🎀 - on
NeighborChangedAddressin ThreeWay finishes in OneWay: no action 🌸 - on
HALChangedin ThreeWay finishes in ThreeWay: store the new HAL 🍓 - on
SendLiein ThreeWay finishes in ThreeWay:SEND_LIE📨 - on
MultipleNeighborsin MultipleNeighborsWait finishes in MultipleNeighborsWait: start multiple neighbors timer with the intervalmultiple_neighbors_lie_holdtime_multiplier * default_lie_holdtime⏲️ - on
FloodLeadersChangedin MultipleNeighborsWait finishes in MultipleNeighborsWait: update you_are_flood_repeater LIE elements based on the flood leader election results 🧁 - on
TimerTickin MultipleNeighborsWait finishes in MultipleNeighborsWait: check MultipleNeighbors timer, if the timer expired, PUSHMultipleNeighborsDone⏰ - on
ValidReflectionin MultipleNeighborsWait finishes in MultipleNeighborsWait: no action 🫶 - on
UpdateZTPOfferin MultipleNeighborsWait finishes in MultipleNeighborsWait: send the offer to the ZTP FSM 🎁 - on
NeighborDroppedReflectionin MultipleNeighborsWait finishes in MultipleNeighborsWait: no action 😮💨 - on
LieRcvdin MultipleNeighborsWait finishes in MultipleNeighborsWait: no action 🙈 - on
UnacceptableHeaderin MultipleNeighborsWait finishes in MultipleNeighborsWait: no action 🙃 - on
NeighborChangedAddressin MultipleNeighborsWait finishes in MultipleNeighborsWait: no action 🏡 - on
LevelChangedin MultipleNeighborsWait finishes in OneWay: update the level with the event value 📈 - on
HATChangedin MultipleNeighborsWait finishes in MultipleNeighborsWait: store HAT 🗂️ - on
MTUMismatchin MultipleNeighborsWait finishes in MultipleNeighborsWait: no action 📏 - on
HALSChangedin MultipleNeighborsWait finishes in MultipleNeighborsWait: store the HALS 🌷 - on
HALChangedin MultipleNeighborsWait finishes in MultipleNeighborsWait: store the new HAL 💾 - on
HoldtimeExpiredin MultipleNeighborsWait finishes in MultipleNeighborsWait: no action 💀 - on
SendLiein MultipleNeighborsWait finishes in MultipleNeighborsWait: no action 🫠 - on
MultipleNeighborsDonein MultipleNeighborsWait finishes in OneWay: no action 🎉 - on Entry into OneWay:
CLEANUP🧹
6.3. Topology Exchange (TIE Exchange) 🦋
6.3.1. Topology Information Elements ✨
Okay so, topology and reachability information in RIFT gets conveyed by TIEs. 🗺️💖
The TIE exchange mechanism uses the port indicated by each node in the LIE exchange as flood_port in LIEPacket and the interface on which the adjacency has been formed as the destination. TIEs MUST be sent with an IPv4 Time to Live (TTL) or an IPv6 Hop Limit (HL) of either 1 or 255 and also MUST be ignored if received with values different than 1 or 255. This helps to protect RIFT information from being accepted beyond a single L3 next hop in the topology. 🛡️✨ TIEs SHOULD be sent with network control precedence unless an implementation is prevented from doing so [RFC2474]. 🫶
TIEs contain sequence numbers, lifetimes, and a type. Each type has ample identifying number space, and information is spread across multiple TIEs with the same TIEElement type (this is true for all TIE types). 🌈 Literally so much room to breathe here 🫧
More information about the TIE structure can be found in the schema in Section 7, starting with TIEPacket root. 📚💜
6.3.2. Southbound and Northbound TIE Representation ✨
okay bestie, this is the part where every node basically wears two outfits 👗👗 depending on which way it's sending its info! 🦋 A spine node keeps two different TIE databases over its adjacencies: one for when it's talking to the north, and one for when it's talking to the south or sideways (east-west). 💅 Those get called southbound and northbound TIEs (South TIEs and North TIEs), based on which direction the stuff is flowing. 🌸
So the North TIEs hold all of the node's adjacencies and local prefixes 🍓, while the South TIEs hold all the adjacencies too, plus the default prefix (with any necessary disaggregated prefixes) and local prefixes. ✨ Section 6.5 has the deeper tea on this one. 🫶
Almost all TIE types are mostly symmetrical in both directions 💜. Section 7.3 is where the TIE types (the TIETypeType element) and their directionality (the _direction_ inside the _TIEID_ element) are defined. 🎀
Here's an example that shows a database holding both representations 🧚. The topology is Figure 2, with the optional link between spine 111 and spine 112 added so we can see flooding on an East-West link 🌈. Unnumbered interfaces are implicitly assumed, and to keep things simple we skip the key value elements that might show up in South or North TIEs 🙈. First up, Figure 15 shows the TIEs generated by some nodes:
ToF 21 South TIEs:
South Node TIE:
NodeTIEElement(level=2,
neighbors(
(Spine 111, level 1, cost 1, links(...)),
(Spine 112, level 1, cost 1, links(...)),
(Spine 121, level 1, cost 1, links(...)),
(Spine 122, level 1, cost 1, links(...))
)
)
South Prefix TIE:
PrefixTIEElement(prefixes(0/0, metric 1), (::/0, metric 1))
Spine 111 South TIEs:
South Node TIE:
NodeTIEElement(level=1,
neighbors(
(ToF 21, level 2, cost 1, links(...)),
(ToF 22, level 2, cost 1, links(...)),
(Spine 112, level 1, cost 1, links(...)),
(Leaf111, level 0, cost 1, links(...)),
(Leaf112, level 0, cost 1, links(...))
)
)
South Prefix TIE:
PrefixTIEElement(prefixes(0/0, metric 1), (::/0, metric 1))
Spine 111 North TIEs:
North Node TIE:
NodeTIEElement(level=1,
neighbors(
(ToF 21, level 2, cost 1, links(...)),
(ToF 22, level 2, cost 1, links(...)),
(Spine 112, level 1, cost 1, links(...)),
(Leaf111, level 0, cost 1, links(...)),
(Leaf112, level 0, cost 1, links(...))
)
)
North Prefix TIE:
PrefixTIEElement(prefixes(Spine 111.loopback)
Spine 121 South TIEs:
South Node TIE:
NodeTIEElement(level=1,
neighbors(
(ToF 21, level 2, cost 1, links(...)),
(ToF 22, level 2, cost 1, links(...)),
(Leaf121, level 0, cost 1, links(...)),
(Leaf122, level 0, cost 1, links(...))
)
)
South Prefix TIE:
PrefixTIEElement(prefixes(0/0, metric 1), (::/0, metric 1))
Spine 121 North TIEs:
North Node TIE:
NodeTIEElement(level=1,
neighbors(
(ToF 21, level 2, cost 1, links(...)),
(ToF 22, level 2, cost 1, links(...)),
(Leaf121, level 0, cost 1, links(...)),
(Leaf122, level 0, cost 1, links(...))
)
)
North Prefix TIE:
PrefixTIEElement(prefixes(Spine 121.loopback)
Leaf112 North TIEs:
North Node TIE:
NodeTIEElement(level=0,
neighbors(
(Spine 111, level 1, cost 1, links(...)),
(Spine 112, level 1, cost 1, links(...))
)
)
North Prefix TIE:
PrefixTIEElement(prefixes(Leaf112.loopback, Prefix112, Prefix_MH))
Figure 15: Example TIEs Generated in a 2-Level Spine-and-Leaf
TopologyIt might not be obvious why the South Node TIEs carry all the adjacencies of the node 🤔. This is going to be necessary for algorithms we dig into more in Sections 6.3.9 and 6.8.7. 🔥
Now, when Node TIEs have more adjacencies than fit in one MTU-sized packet 📦, the _neighbors_ element can hold a different set of neighbors in each TIE. Those disjointed sets of neighbors MUST be joined during the corresponding computation 🧩. However, if any of the following happens across multiple Node TIEs:
- _capabilities_ do not match or
- _flags_ values do not match or
- the same neighbor repeats in multiple TIEs with different values.
...then the implementation is expected to use the value from any of the valid TIEs it received 🤷♀️, because it can't control the order the TIEs arrive in. 🌀
The _miscabled_links_ element SHOULD be included in every Node TIE 💌. If it isn't, the behavior is undefined 💀.
A ToF node MUST include information on all other ToFs it knows about through reflection 👑. The _same_plane_tofs_ element is what carries this info. To avoid MTU overrun problems, multiple Node TIEs can carry disjointed sets of ToFs, and those MUST be joined to form a single set 🎀.
Different TIE types are carried in _TIEElement_ 🦋. The schema enum 'common.TIETypeType' in _TIEID_ tells you which elements MUST be present in _TIEElement_. If the _TIETypeType_ in the _TIEID_ doesn't match the element that's actually there, the unexpected elements MUST be ignored 🙃. If an expected element is missing from the TIE, an error MUST be reported and the TIE MUST be ignored 😭. The _positive_disaggregation_prefixes_ and _positive_external_disaggregation_prefixes_ elements MUST be advertised southbound only and ignored in North TIEs 🌸. The _negative_disaggregation_prefixes_ element MUST be propagated southwards towards lower levels, according to Section 6.5.2, to heal pathological upper-level partitioning 🩹; otherwise, traffic loss may occur in multi-plane fabrics 🥺. It MUST NOT be advertised within a North TIE, and it MUST be ignored otherwise 🚫.
6.3.3. Flooding 🌊
As we covered before, TIEs get transported over UDP using the ports indicated in the LIE exchanges, and they go to the destination address the LIE adjacency was formed on 📬.
TIEs are uniquely identified by the _TIEID_ schema element 🪪. _TIEID_ gives you a total order by comparing its elements in the order they're defined, with each value compared as an unsigned integer of the matching length 🔢. The _TIEHeader_ element has a _seq_nr_ element that tells newer versions of the same TIE apart 🆕.
_TIEHeader_ can also carry an _origination_time_ schema element (for fabrics that use precision timing ⏱️), which holds the absolute timestamp of when the TIE was generated. It can also carry an _origination_lifetime_, which shows the original lifetime when the TIE was made 🕰️. When these are included, they can be used for debugging or security purposes (for example, to prevent lifetime modification attacks 🛡️). Clock synchronization gets a closer look in Section 6.8.4 ⏰.
_remaining_lifetime_ counts down to 0 from _origination_lifetime_ ⏳. TIEs whose lifetimes differ by less than _lifetime_diff2ignore_ MUST be considered EQUAL (as long as all their other fields are equal too) 👯. This constant MUST be larger than _purge_lifetime_ so that we avoid retransmissions 🔁.
This normative ordering methodology is described in Figure 16 and MUST be used by all implementations 📜.
function Compare(X: TIEHeader, Y: TIEHeader) returns Ordering:
seq_nr of a TIEHeader = TIEHeader.seq_nr
TIEID of a TIEHeader = TIEHeader.TIEID
direction of a TIEID = TIEID.direction
# System ID
originator of a TIEID = TIEID.originator
# is of type TIETypeType
tietype of a TIEID = TIEID.tietype
tie_nr of a TIEID = TIEID.tie_nr
if X.direction > Y.direction:
return X is larger
else if X.direction < Y.direction:
return Y is larger
else if X.originator > Y.originator:
return X is larger
else if X.originator < Y.originator:
return Y is larger
else:
if X.tietype == Y.tietype:
if X.tie_nr == Y.tie_nr:
if X.seq_nr == Y.seq_nr:
X.lifetime_left = X.remaining_lifetime
- time since TIE was received
Y.lifetime_left = Y.remaining_lifetime
- time since TIE was received
if absolute_value_of(X.lifetime_left -
Y.lifetime_left) <= common.lifetime_diff2ignore:
return Both are Equal
else:
return TIEHeader with larger lifetime_left is
larger
else:
return TIEHeader with larger seq_nr is larger
else:
return TIEHeader with larger tie_nr is larger
else:
return TIEHeader with larger TIEType is largerFigure 16: TIEHeader Comparison Function 🧮
All valid TIE types are defined in _TIETypeType_ 🗂️. This enum says what TIE type a TIE is carrying. If the value isn't known to the receiver, the TIE MUST be reflooded with the scope identical to the scope of a prefix TIE 🔁. That leaves room for future extensions of the protocol that stay within the same major schema and have types that are opaque to some nodes 🔮. Some restrictions are defined in Section 7 📖.
6.3.3.1. Normative Flooding Procedures 🧠
On reception of a TIE with an undefined level value in the packet header, the node MUST issue a warning and discard the packet ⚠️.
This section specifies the precise, normative flooding mechanism. You can skip it unless you're building an implementation of the protocol or want a deep understanding of the underlying information distribution mechanism 🤓.
Flooding procedures are described in terms of the flooding state of an adjacency, and the resulting operations on it are driven by packet arrivals 📨. Implementations MUST implement a behavior that is externally indistinguishable from the FSMs and normative procedures given here 🪞.
RIFT does not specify any kind of flood rate limiting 🚦. To help adjust flooding speeds, the encoded packets give hints for reacting to losses or overruns: _you_are_sending_too_quickly_ in the _LIEPacket_, and "Packet Number" in the security envelope described in Section 6.9.3 🫧. Flooding of all corresponding topology exchange elements SHOULD be performed at the highest feasible rate 🏎️, but the transmission rate MUST be throttled by reacting to packet elements and features of the system, such as queue lengths or congestion indications in the protocol packets 🚥.
A node SHOULD NOT send out any topology information elements if the adjacency is not in a _ThreeWay_ state 🙅♀️. No further tightening of this rule is possible 🤷♀️. For example, link buffering may cause both LIEs and TIEs/TIDEs/TIREs to be reordered 🌪️.
A node MUST drop any received TIEs/TIDEs/TIREs unless it is in the _ThreeWay_ state 🗑️.
TIEs generated by other nodes MUST be reflooded 🔁. TIDEs and TIREs MUST NOT be reflooded 🚫.
6.3.3.1.1. FloodState Structure per Adjacency ✨
Okay bestie, let's talk about what each adjacency is carrying around 🎒 For every adjacency, the structure conceptually holds these elements. "Collection" or "queue" just means a set of things you can loop over 🔁
TIES_TX: every TIE that needs to go out on the adjacency 📤TIES_ACK: every TIE that has to be acknowledged on the adjacency 🙏TIES_REQ: every TIE header that has to be requested on the adjacency 🙋TIES_RTX: every TIE that needs retransmission, along with its retransmit time ⏰FILTERED_TIEDB: a filtered view of TIEDB that only keeps the headersis_tide_entry_filteredallows, and that either still have a lifetime left > 0 or have no content 🧹
Now the well-known words and procedures that work on this structure 💅
- TIE: this can mean either a full RIFT TIE or just the TIEHeader or TIEID equivalent, as defined in Section 7.3. Each algorithm makes it clear which one it means 🧠
is_flood_reduced(TIE): tells you whether a TIE can be flood-reduced or not 🌿is_tide_entry_filtered(TIE): tells you whether a header should be propagated in a TIDE, based on flooding scopes 🎯is_request_filtered(TIE): tells you whether a TIE request should be propagated to the neighbor, based on flooding scopes 🚦is_flood_filtered(TIE): tells you whether a requested TIE should be flooded to the neighbor, based on flooding scopes 🌊
try_to_transmit_tie(TIE) is the one that does the heavy lifting, so pay attention 👀
- If not
is_flood_filtered(TIE), then:- Remove the TIE from
TIES_RTXif it's there 🗑️ - If a TIE with the same key is found on
TIES_ACK, then:- a. If the TIE is the same as or newer than TIE, do nothing 🙅, else
- b. Remove the TIE from
TIES_ACKand add TIE toTIES_TX📦
- Else, insert the TIE into
TIES_TX➕
- Remove the TIE from
Here are the rest of the moves in the playbook 📖
ack_tie(TIE): remove the TIE from all collections, then insert it intoTIES_ACK✅tie_been_acked(TIE): remove the TIE from all collections 👋remove_from_all_queues(TIE): same astie_been_ackedrequest_tie(TIE): if notis_request_filtered(TIE), then callremove_from_all_queues(TIE)and add the TIE toTIES_REQ🙋♀️move_to_rtx_list(TIE): remove the TIE fromTIES_TX, then add it toTIES_RTXusing the TIE retransmission interval ⏳clear_requests(TIEs): remove all TIEs fromTIES_REQ🧽bump_own_tie(TIE): for a self-originated TIE, originate an empty one or regenerate it with a version number higher than the one in the TIE 🆙
If the system can't process all the collections in real time, the collections SHOULD be served in these priorities 🚨
- Elements on
TIES_ACKshould be processed with the highest priority 🥇 TIES_TX🥈TIES_REQandTIES_RTXshould be processed with the lowest priority 🥉
6.3.3.1.2. TIDEs 🍯
Okay, so the TIEID and TIEHeader spaces form a strict total order 📏 There's one tiny exception, though. Sequence numbers (found in TIEHeader.seq_nr) can become incomparable, as explained in Appendix A 🌀 That only happens in a very unlikely situation: a TIE gets "stuck" somewhere in the network while its originator reboots and reissues TIEs so many times that the sequence number rolls over and ends up an incomparable distance away from the "stuck" copy 😱 Other than that, you can compare any two elements. That means TIEs, TIEHeaders, and TIEIDs can all be compared with each other, and the shortest viable key is always implied 💜
6.3.3.1.2.1. TIDE Generation 🎀
NEXT_TIDE_ID is the ID of the next TIE to be sent in the TIDE 🎯
As set by the timer constant, TIDEs get generated periodically by doing this 🔄
NEXT_TIDE_ID = MIN_TIEID- While
NEXT_TIDE_IDis not equal toMAX_TIEID, do:- a.
HEADERS= exactlyTIRES_PER_TIDE_PKTheaders fromFILTERED_TIEDBstarting atNEXT_TIDE_ID, unless fewer thanTIRES_PER_TIDE_PKTremain, in which case, all the remaining headers 🧺 - b. If
HEADERSis empty, thenSTART = MIN_TIEID, elseSTART= first element inHEADERS🏁 - c. If the
HEADERSsize is less thanTIRES_PER_TIDE_PKT, thenEND = MAX_TIEID, elseEND= last element inHEADERS🏳️ - d. Send sorted
HEADERSas a TIDE, withSTARTandENDas its range 📬 - e.
NEXT_TIDE_ID = END⏭️
- a.
The constant TIRES_PER_TIDE_PKT SHOULD be computed per interface. The implementation uses it to cap how many TIE headers go in each TIDE, so the sent TIDE PDU doesn't exceed the MTU of the interface 📏💖
TIDE PDUs SHOULD be transmitted at a rate that does not lead to packet drops 🐢
The algorithm intentionally goes through the loop once and sends a single TIDE, even when the database is empty. Otherwise, no TIDEs would be sent for an empty database, and that would break the intended synchronization 🙈
6.3.3.1.2.2. TIDE Processing 🧩
Here's the cast of characters for TIDE processing 🎭
TXKEYS: collection of TIE headers to be sent after the packet is processed 📤REQKEYS: collection of TIEIDs to be requested after the packet is processed 🙋CLEARKEYS: collection of TIEIDs to be removed from flood state queues 🧹LASTPROCESSED: last processed TIEID in the TIDE 📍DBTIE: the TIE in the LSDB, if one is found 🔍
When a TIDE comes in, here's the processing 🫶
LASTPROCESSED = TIDE.start_range- For every
HEADERin the TIDE, do:- a.
DBTIE= findHEADERin the current LSDB 🕵️ - b. If
HEADER<LASTPROCESSED, then report an error, reset the adjacency, and return 🚨 - c. Put all TIEs in the LSDB where (
TIE.HEADER>LASTPROCESSEDandTIE.HEADER<HEADER) intoTXKEYS📦 - d.
LASTPROCESSED = HEADER📍 - e. If
DBTIEis not found, then:- i. If the originator is this node, then
bump_own_tie🆙 - ii. Else, put
HEADERintoREQKEYS🙋
- i. If the originator is this node, then
- f. If
DBTIE.HEADER<HEADER, then:- i. If the originator is this node, then
bump_own_tie, else:- 1. If this is a North TIE header from a northbound neighbor, then override
DBTIEin the LSDB withHEADER🔁 - 2. Else, put
HEADERintoREQKEYS🙋
- 1. If this is a North TIE header from a northbound neighbor, then override
- i. If the originator is this node, then
- g. If
DBTIE.HEADER>HEADER, then putDBTIE.HEADERintoTXKEYS📤 - h. If
DBTIE.HEADER=HEADER, then:- i. If
DBTIEalready has content, then putDBTIE.HEADERintoCLEARKEYS🧹, else: - ii. Put
HEADERintoREQKEYS🙋
- i. If
- a.
- Put all TIEs in the LSDB where (
TIE.HEADER>LASTPROCESSEDandTIE.HEADER<=TIDE.end_range) intoTXKEYS📦 - For all TIEs in
TXKEYS, calltry_to_transmit_tie(TIE)📤 - For all TIEs in
REQKEYS, callrequest_tie(TIE)🙋 - For all TIEs in
CLEARKEYS, callremove_from_all_queues(TIE)🧽
6.3.3.1.3. TIREs 🌷
6.3.3.1.3.1. TIRE Generation 🌸
Elements from both TIES_REQ and TIES_ACK MUST be collected and sent out as fast as feasible as TIREs 🚀 When you send TIREs with elements from TIES_REQ, the remaining_lifetime field in TIEHeaderWithLifeTime MUST be set to 0. That forces the neighbor to reflood even if the TIEs look the same 🔁✨
6.3.3.1.3.2. TIRE Processing 🌼
Here's the cast for TIRE processing 🎭
TXKEYS: collection of TIE headers to be sent after the packet is processed 📤REQKEYS: collection of TIEIDs to be requested after the packet is processed 🙋ACKKEYS: collection of TIEIDs that have been acknowledged ✅DBTIE: the TIE in the LSDB, if one is found 🔍
When TIREs come in, do this 💅
- For every
HEADERin the TIRE, do:- a.
DBTIE= findHEADERin the current LSDB 🕵️ - b. If
DBTIEis not found, then do nothing 🤷 - c. If
DBTIE.HEADER<HEADER, then putHEADERintoREQKEYS🙋 - d. If
DBTIE.HEADER>HEADER, then putDBTIE.HEADERintoTXKEYS📤 - e. If
DBTIE.HEADER=HEADER, then putDBTIE.HEADERintoACKKEYS✅
- a.
- For all TIEs in
TXKEYS, calltry_to_transmit_tie(TIE)📤 - For all TIEs in
REQKEYS, callrequest_tie(TIE)🙋 - For all TIEs in
ACKKEYS, calltie_been_acked(TIE)👋
6.3.3.1.4. TIEs Processing on Flood State Adjacency 🌊
When TIEs come in, here's the processing 🫧
Here's the cast for this one 🎭
ACKTIE: TIE to acknowledge 🙏TXTIE: TIE to transmit 📤DBTIE: the TIE in the LSDB, if one is found 🔍
DBTIE= find TIE in the current LSDB 🕵️- If
DBTIEis not found, then:- a. If the originator is this node, then
bump_own_tiewith a short remaining lifetime ⏳ - b. Else, insert TIE into the LSDB and set
ACKTIE = TIE📥
- a. If the originator is this node, then
Else:
- a. If
DBTIE.HEADER=TIE.HEADER, then:- i. If
DBTIEalready has content, thenACKTIE = TIE✅ - ii. Else, process it like the "
DBTIE.HEADER<TIE.HEADER" case 🔁
- i. If
- b. If
DBTIE.HEADER<TIE.HEADER, then:- i. If the originator is this node, then
bump_own_tie🆙 - ii. Else, insert TIE into the LSDB and set
ACKTIE = TIE📥
- i. If the originator is this node, then
- c. If
DBTIE.HEADER>TIE.HEADER, then:- i. If
DBTIEalready has content, thenTXTIE = DBTIE📤 - ii. Else,
ACKTIE = DBTIE✅
- i. If
- If
TXTIEis set, then calltry_to_transmit_tie(TXTIE)📤 - If
ACKTIEis set, then callack_tie(TIE)🙏
6.3.3.1.5. Sending TIEs 📮
On a periodic basis, every TIE with a lifetime of > 0 left MUST be sent out on the adjacency 💌 Each one then gets removed from the TIES_TX list and requeued onto the TIES_RTX list 🔁 The specific period is out of scope for this document 🙋♀️
6.3.3.1.6. TIEs Processing in LSDB 🗄️
The LSDB holds the most recent copy of TIEs received via flooding from the corresponding peers 💾 After version tie-breaking by the LSDB, a peer gets the newest versions of TIEs that other peers received from the LSDB, and processes them (without any filtering) just like TIEs from its remote peer 📬 That's a publisher model, and it can be implemented in several ways, either in a single thread of execution or in multiple parallel threads 🧵
The LSDB can be logically considered the entity that ages out TIEs 👵 It's responsible for discarding TIEs that have been stored longer than remaining_lifetime on their reception ⏰
The LSDB is also expected to periodically reoriginate the node's own TIEs 🔄 Reoriginating at an interval significantly shorter than default_lifetime is RECOMMENDED. That stops other nodes in the network from expiring the TIEs, which can lead to instabilities 🫠✨
6.3.4. TIE Flooding Scopes ✨
Okay so, kinda like the link-local, area, and domain flooding scopes 🦋, RIFT has a few extra-complex "flooding scopes" 💅. Which scope a TIE gets depends on its direction and its type.
Every North TIE gets flooded northbound 🚀, so a node at a given level ends up with the full topology of the Clos or fat tree network that's reachable southwards of it, specific prefixes included 🌸. The payoff is that if a packet comes in from a node at the same or a lower level and its destination is covered by one of those specific prefixes, it gets routed straight to the node advertising that prefix, instead of getting bounced up to a higher level. Iconic efficiency 👑.
A node's South Node TIEs are its adjacencies, plus South Prefix TIEs that are limited to the default IP prefix and disaggregated prefixes 🧚. These get flooded southbound to tell the nodes one level down about the connectivity of the higher level and how to reach the rest of the fabric. Now here's the tea 🫖: to let an E-W disconnected node at a level get the South TIEs of the other nodes at its level, every South Node TIE gets "reflected" northbound back to the level it came from 🪞. East-West links are included in South TIE flooding too, except at the ToF level 💜. Those TIEs have to be flooded so the algorithms in Section 6.4 can work. This way nodes at the same level can learn about each other without going through a lower level, except at the leaf level 🍓. The precise, normative flooding scopes live in Table 3, and those same rules also decide what SHOULD be included in TIDEs on the adjacency. East-West flooding scopes are basically the same as the southern ones, except for ToF East-West links (rings), which are mostly doing northbound flooding 🌈.
South Node TIE "south reflection" is what makes positive disaggregation on failures work (see Section 6.5) 💀, and it's also part of flooding reduction (see Section 6.3.9) ✨.
+===========+======================+==============+=================+
| Type / | South | North | East-West |
| Direction | | | |
+===========+======================+==============+=================+
| South | flood if the level | flood if the | flood only if |
| Node TIE | of the originator | level of the | this node is |
| | is equal to this | originator | not ToF |
| | node | is higher | |
| | | than this | |
| | | node | |
+-----------+----------------------+--------------+-----------------+
| non-Node | flood self- | flood only | flood only if |
| South TIE | originated only | if the | it is self- |
| | | neighbor is | originated and |
| | | the | this node is |
| | | originator | not ToF |
| | | of TIE | |
+-----------+----------------------+--------------+-----------------+
| all North | never flood | flood always | flood only if |
| TIEs | | | this node is |
| | | | ToF |
+-----------+----------------------+--------------+-----------------+
| TIDE | include at least | include at | if this node is |
| | all non-self- | least all | ToF, then |
| | originated North | South Node | include all |
| | TIE headers and | TIEs and all | North TIEs; |
| | self-originated | South TIEs | otherwise, only |
| | South TIE headers | originated | include self- |
| | and South Node TIEs | by a peer | originated TIEs |
| | of nodes at same | and all | |
| | level | North TIEs | |
+-----------+----------------------+--------------+-----------------+
| TIRE as | request all North | request all | if this node is |
| Request | TIEs and all peer's | South TIEs | ToF, then apply |
| | self-originated | | north scope |
| | TIEs and all South | | rules; |
| | Node TIEs | | otherwise, |
| | | | apply south |
| | | | scope rules |
+-----------+----------------------+--------------+-----------------+
| TIRE as | Ack all received | Ack all | Ack all |
| Ack | TIEs | received | received TIEs |
| | | TIEs | |
+-----------+----------------------+--------------+-----------------+Table 3: Normative Flooding Scopes
If the TIDE includes additional TIE headers beyond the ones specified, the receiving neighbor must apply the matching filter to the received TIDE strictly 🔍, and it MUST NOT request the extra TIE headers that the flooding scope rules didn't allow in its direction 🙅♀️.
To show you how this all plays out 🎀, check out the topology in Figure 2 with the optional link between spine 111 and spine 112, plus the TIEs that go with it in Figure 15. The flooding from specific nodes is laid out in Table 4 👀.
+============+==========+===========================================+
| Local | Neighbor | TIEs Flooded from Local to Neighbor Node |
| Node | Node | |
+============+==========+===========================================+
| Leaf111 | Spine | Leaf111 North TIEs, Spine 111 South Node |
| | 112 | TIE |
+------------+----------+-------------------------------------------+
| Leaf111 | Spine | Leaf111 North TIEs, Spine 112 South Node |
| | 111 | TIE |
+------------+----------+-------------------------------------------+
| ... | ... | ... |
+------------+----------+-------------------------------------------+
| Spine | Leaf111 | Spine 111 South TIEs |
| 111 | | |
+------------+----------+-------------------------------------------+
| Spine | Leaf112 | Spine 111 South TIEs |
| 111 | | |
+------------+----------+-------------------------------------------+
| Spine | Spine | Spine 111 South TIEs |
| 111 | 112 | |
+------------+----------+-------------------------------------------+
| Spine | ToF 21 | Spine 111 North TIEs, Leaf111 North TIEs, |
| 111 | | Leaf112 North TIEs, ToF 22 South Node TIE |
+------------+----------+-------------------------------------------+
| Spine | ToF 22 | Spine 111 North TIEs, Leaf111 North TIEs, |
| 111 | | Leaf112 North TIEs, ToF 21 South Node TIE |
+------------+----------+-------------------------------------------+
| ... | ... | ... |
+------------+----------+-------------------------------------------+
| ToF 21 | Spine | ToF 21 South TIEs |
| | 111 | |
+------------+----------+-------------------------------------------+
| ToF 21 | Spine | ToF 21 South TIEs |
| | 112 | |
+------------+----------+-------------------------------------------+
| ToF 21 | Spine | ToF 21 South TIEs |
| | 121 | |
+------------+----------+-------------------------------------------+
| ToF 21 | Spine | ToF 21 South TIEs |
| | 122 | |
+------------+----------+-------------------------------------------+
| ... | ... | ... |
+------------+----------+-------------------------------------------+Table 4: Flooding Some TIEs from Example Topology
6.3.5. RAIN: RIFT Adjacency Inrush Notification 🌊
The optional RIFT Adjacency Inrush Notification (RAIN) mechanism is there to keep adjacencies from getting totally swamped by flooding when a node restarts or comes up with a bunch of southbound neighbors 😭. In its LIEs, a node MAY set the _you_are_sending_too_quickly_ flag 🚩. That tells the neighbor it SHOULD flood Node TIEs at normal speed and really slow down the flooding of every other TIE 🐢. The flag SHOULD be set only in the southbound direction 🧭. The receiving node SHOULD accommodate the request and lighten the flooding load on the affected node if that node is south of the sender 🫶, and it should ignore the indication if the node is north of the sender.
Keeping Node TIEs at normal speed, even under heavy load, is what guarantees correct behavior for algorithms like disaggregation or default route origination 🧠. That said, there's a real trade-off with this bit 💅: processing load goes down, but convergence speed can suffer, since slowing down flooding of northbound prefixes from neighbors for a long time will lead to traffic losses 💔.
6.3.6. Initial and Periodic Database Synchronization 🔄
The initial exchange in RIFT includes periodic TIDE exchanges 💌. Those carry descriptions of the LSDB and TIREs, and they do two jobs: they request TIEs you don't have yet, and they confirm that flooded TIEs were actually received 🤝. What goes into TIDEs and TIREs is governed by Table 3 ✨.
6.3.7. Purging and Rollovers 🧹✨
Okay so when a node exits the network and it's "unpurged", residual stale TIEs can hang around until their lifetimes expire. In RIFT that's a rather long period by default 😅, which stops TIEs from getting reoriginated over and over in very large topologies 🌍. RIFT has no "purging mechanism" that sends specialized "purge" packets 🙅♀️. Other routing protocols tried that, and after many years of experience it turned out to be complex and fragile 💀. So RIFT just issues a new, i.e., higher sequence number, empty version of the TIE with a short lifetime given by the _purge_lifetime_ constant. Then every node ages out and deletes each TIE copy on its own 🧹. Abundant memory is available today, even on low-end platforms, so keeping those relatively short-lived extra copies around for a while is totally acceptable 💅. The info ages out, and in the meantime all computations still deliver correct results if a node leaves, because the new information from its adjacent nodes breaks bidirectional connectivity checks in different computations ✨.
Once a RIFT node issues a TIE with an ID, it SHOULD preserve that ID as long as feasible 🆔 (also when the protocol restarts), even if the TIE loses all its content. The re-advertisement of an empty TIE does the job of purging any information advertised in previous versions 🫧. The originator is free to skip reoriginating that empty TIE again, or to originate an empty TIE with a relatively short lifetime so a big pile of long-lived empty stubs doesn't pollute the network 🙈. Each node MUST time out and clean up the corresponding empty TIEs independently ⏰.
Upon restart, a node MUST be prepared to receive TIEs with its own System ID and supersede them with equivalent, newly generated, empty TIEs with a higher sequence number 🔄. As above, the lifetime can be relatively short, since it only needs to exceed the necessary propagation and processing delay of all the nodes that are within the TIE's flooding scope 🚀.
TIE sequence numbers are rolled over using the method described in Appendix A 🎀. The first sequence number of any spontaneously originated TIE (i.e., not originated to override a detected older copy in the network) MUST be a reasonably unpredictable random number (for example, [RFC4086]) in the interval [0, 2^30-1]. This stops otherwise identical TIE headers from staying "stuck" in the network with content that's different from the TIE originated after reboot 👀. In typical link-state protocols, this job is handed to a 16-bit checksum on the packet content. RIFT skips that design because of the CPU burden of computing such checksums 🥵. Plus the checksum has to be "patched" into the packet after the content is generated, which is already a difficult move in binary, hand-crafted formats and is highly incompatible with model-based, serialized formats 🙃. So the sequence number space is consciously chosen to be 64-bits wide 🔢. That makes a TIE with the same sequence number but different content as much or even more unlikely than the checksum method. To emulate the "checksum behavior", an implementation could choose to compute a 64-bit checksum or hash function over the TIE content and use that as part of the first sequence number after reboot 🧩.
6.3.8. Southbound Default Route Origination 🛣️
Under certain conditions, nodes issue a default route in their South Prefix TIEs, with costs as computed in Section 6.8.7.1 💸.
A node X that:
- is not overloaded and
- has southbound or East-West adjacencies
SHOULD originate such a default route in its South Prefix TIE if and only if:
- all other nodes at X's level are overloaded or
- all other nodes at X's level have NO northbound adjacencies, or
- X has computed reachability to a default route during N-SPF.
The term "all other nodes at X's level" obviously describes just the nodes at the same level in the PoD that have a viable lower level 🏘️. (Otherwise, the South Node TIEs cannot be reflected, and the nodes in PoD 1 and PoD 2 are "invisible" to each other 👻.)
A node originating a southbound default route SHOULD install a default discard route if it did not compute a default route during N-SPF 🗑️. This basically means the top of the fabric will drop traffic for unreachable addresses 🚫. Bye bye, lost packets 💅.
6.3.9. Northbound TIE Flooding Reduction 🌊
RIFT chooses only a subset of northbound nodes to propagate flooding 🎯. That balances the flooding across the fabric (so no "hot" flooding links 🔥) and also reduces its volume. The solution is based on several principles:
- a node MUST flood self-originated North TIEs to all the reachable nodes at the level above, which is called the node's "parents";
- it is typically not necessary that all parents reflood the North TIEs to achieve a complete flooding of all the reachable nodes two levels above, which we call the node's "grandparents";
- to control the volume of its flooding two hops north and yet keep it robust enough, it is advantageous for a node to select a subset of its parents as "Flood Repeaters" (FRs), which when combined, deliver two or more copies of its flooding to all of its parents, i.e., the originating node's grandparents 🔁;
- nodes at the same level do not have to agree on a specific algorithm to select the FRs, but overall load balancing should be achieved so that different nodes at the same level should tend to select different parents as FRs (consideration of possible strategies in an unrelated but similar field can be found in [RFC2991]) ⚖️;
- there are usually many solutions to the problem of finding a set of FRs for a given node. The problem of finding the minimal set is (similar to) an NP-Complete problem, and a globally optimal set may not be the minimal one if load balancing with other nodes is an important consideration 🧠;
- it is expected that sets of equivalent nodes at a level L will often exist, defined as having a common set of parents at L+1. Applying this observation at both L and L+1, an algorithm may attempt to split the larger problem into a sum of smaller, separate problems 🧩; and
- it is expected that there will be a broken link between a parent and a grandparent from time to time, and in that case, the parent is probably a poor FR due to its lower reliability 💔. An algorithm may attempt to eliminate parents with broken northbound adjacencies first, in order to reduce the number of FRs. Albeit it could be argued that relying on higher fanout FRs will slow flooding due to higher replication, the reliability of the FR's links is likely a more pressing concern 🔗.
In a fully connected Clos network, this means a node selects one arbitrary parent as the FR and then a second one for redundancy 🤝. The computation can be relatively simple and completely distributed, without any need for synchronization among nodes ✨. In a "PoD" structure, where level L+2 is partitioned into silos of equivalent grandparents that are only reachable from their respective parents, this means treating each silo as a fully connected Clos network and solving the problem within the silo 🏰.
In terms of signaling, a node has enough information to select its set of FRs 📡. This information is derived from the node's parents' South Node TIEs, which indicate the parent's reachable northbound adjacencies to its own parents (the node's grandparents). A node may send a LIE to a northbound neighbor with the optional boolean field _you_are_flood_repeater_ set to false, to indicate that the northbound neighbor is not a flood repeater for the node that sent the LIE 🙅. In that case, the northbound neighbor SHOULD NOT reflood northbound TIEs received from the node that sent the LIE. If _you_are_flood_repeater_ is absent or _you_are_flood_repeater_ is set to true, then the northbound neighbor is a flood repeater for the node that sent the LIE and MUST reflood northbound TIEs received from that node 🫶. The element _you_are_flood_repeater_ MUST be ignored if received from a northbound adjacency 🚫.
This specification provides a simple default algorithm that SHOULD be implemented and used by default on every RIFT node 💖. Here's the setup:
- let
|NA(Node)be the set of northbound adjacencies of nodeNodeandCN(Node)be the cardinality of|NA(Node); - let
|SA(Node)be the set of southbound adjacencies of nodeNodeandCS(Node)be the cardinality of|SA(Node); - let
|P(Node)be the set of nodeNode's parents; - let
|G(Node)be the set of nodeNode's grandparents. Observe that|G(Node) = |P(|P(Node)); - let N be the child node at level L computing a set of FRs;
- let P be a node at level L+1 and a parent node of N, i.e., bidirectionally reachable over adjacency
ADJ(N, P); - let G be a grandparent node of N, reachable transitively via a parent P over adjacencies
ADJ(N, P)andADJ(P, G). Observe that N does not have enough information to check bidirectional reachability ofADJ(P, G)🕵️; - let R be a redundancy constant integer; a value of 2 or higher for R is RECOMMENDED 🔐;
- let S be a similarity constant integer; a value in range
0 .. 2for S is RECOMMENDED, and the value of 1 SHOULD be used. Two cardinalities are considered as equivalent if their absolute difference is less than or equal to S, i.e.,|a-b|<=S👯; - let RND be a 64-bit random number (for example, as described in [RFC4086]) generated by the system once on startup 🎲.
The algorithm consists of the following steps 🧮:
- Derive a 64-bit number by XORing N's System ID with RND.
- Derive a 16-bit pseudo-random unsigned integer PR(N) from the resulting 64-bit number by splitting it into 16-bit-long words W1, W2, W3, W4 (where W1 are the least significant 16 bits of the 64-bit number, and W4 are the most significant 16 bits), and then XORing the circularly shifted resulting words together:
- A.
(W1<<1) xor (W2<<2) xor (W3<<3) xor (W4<<4);where<<is the circular shift operator.
- A.
- Sort the parents by decreasing number of northbound adjacencies (using decreasing System ID of the parent as a tie-breaker): sort
|P(N)by decreasingCN(P), for allPin|P(N), as the ordered array|A(N). - Partition
|A(N)in subarrays|A_k(N)of parents with equivalent cardinality of northbound adjacencies (in other words, with equivalent number of grandparents they can reach):
a. set k=0; // k is the ID of the subarray
b. set i=0;
c. while i < CN(N) do
i. set j=i;
ii. while i < CN(N) and CN(|A(N)[j]) - CN(|A(N)[i]) <= S:
1. place |A(N)[i] in |A_k(N) // abstract action, maybe
noop
2. set i=i+1;
iii. /* At this point, j is the index in |A(N) of the first
member of |A_k(N) and (i-j) is C_k(N) defined as the
cardinality of |A_k(N). */
set k=k+1;
/* At this point, k is the total number of subarrays, initialized
for the shuffling operation below. */- Shuffle each subarrays
|A_k(N)of cardinalityC_k(N)within|A(N)individually, using the Durstenfeld variation of the Fisher-Yates algorithm that depends on N's System ID 🔀:
a. while k > 0 do
i. for i from C_k(N)-1 to 1 decrementing by 1 doset j to PR(N) modulo i;🔁exchange |A_k[j] and |A_k[i];💫
- ii.
set k=k-1;
- For each grandparent G, initialize a counter c(G) with the number of its southbound adjacencies to elected flood repeaters (which is initially zero): 🧮
- a. for each G in
|G(N), setc(G) = 0;
- a. for each G in
- Finally, only keep FRs as parents that are needed to maintain the number of adjacencies between the FRs and any grandparent G equal or above the redundancy constant R: 🎯
- a. for each P in reshuffled
|A(N):- i. if there exists an adjacency
ADJ(P, G)in|NA(P)such thatc(G) < R, then- place P in FR set;
- for all adjacencies
ADJ(P, G')in|NA(P)incrementc(G')
- i. if there exists an adjacency
- a. for each P in reshuffled
- If any
c(G)is still< R, it was not possible to elect a set of FRs that covers all grandparents with redundancyR. 🙈
Additional rules for flooding reduction: these are the vibes-check rules for how the flooding reduction actually behaves 💅
- The algorithm MUST be re-evaluated by a node on every change of local adjacencies or reception of a parent South TIE with changed adjacencies. 🔄 A node MAY apply a hysteresis to prevent an excessive amount of computation during periods of network instability, just like in the case of reachability computation. 🧘♀️
- Upon a change of the flood repeater set, a node SHOULD send out LIEs that grant flood repeater status to newly promoted nodes before it sends LIEs that revoke the status to the nodes that have been newly demoted. 🎀 This is done to prevent a transient behavior where the full coverage of grandparents is not guaranteed. Such a condition is sometimes unavoidable in case of lost LIEs, but it will correct itself, at the cost of a possible transient reduction in flooding propagation speeds. ⏳ The election can use the LIE FSM
_FloodLeadersChanged_event to notify LIE FSMs of the necessity to update the sent LIEs. 📣 - A node MUST always flood its self-originated TIEs to all its neighbors. 💖
- A node receiving a TIE originated by a node for which it is not a flood repeater SHOULD NOT reflood such TIEs to its neighbors, except for the rules described in Section 6.3.9, Paragraph 10, Item 6. 🙅♀️
- The indication of flood reduction capability MUST be carried in the Node TIEs in the
_flood_reduction_element and MAY be used to optimize the algorithm to account for nodes that will flood regardless. ✨ - A node generates TIDEs as usual, but when receiving TIREs or TIDEs resulting in requests for a TIE of which the newest received copy came on an adjacency where the node was not a flood repeater, it SHOULD ignore such requests on only the first request. 🫶 Normally, the nodes that received the TIEs as flooding repeaters should satisfy the requesting node and, with that, no further TIREs for such TIEs will be generated. Otherwise, the next set of TIDEs and TIREs MUST lead to flooding independent of the flood repeater status. 🔥 This solves a very difficult "incast" problem on nodes restarting with a very wide fanout, especially northbound. To retrieve the full database, they often end up processing many inrushing copies, whereas this approach load balances the incoming database between adjacent nodes and flood repeaters and should guarantee that two copies are sent by different nodes to ensure against any losses. 🧺
6.3.10. Special Considerations 🎀
First, due to the distributed, asynchronous nature of ZTP, it can create temporary convergence anomalies where nodes at higher levels of the fabric temporarily become lower than where they ultimately belong. 🌀 Flooding can begin before ZTP is "finished", and in fact it must do so, given there is no global termination criteria for the unsynchronized ZTP algorithm. So information may temporarily end up in the wrong layers. A special clause when changing level takes care of that. 🧹
More difficult is a condition where a node (e.g., a leaf) floods a TIE north towards its grandparent, then its parent reboots, partitioning the grandparent from the leaf directly, and then the leaf itself reboots. 😭 That can leave the grandparent holding the "primary copy" of the leaf's TIE. Normally this is resolved easily by the leaf reoriginating its TIE with a higher sequence number than it notices in the northbound TIEs. Here, however, when the parent comes back, it won't be able to obtain the leaf's North TIE from the grandparent easily. So the leaf may not issue the TIE with a higher sequence number that can reach the grandparent for a long time. 🙈 Flooding procedures are extended to deal with the problem through special clauses that override the database of a lower level with headers of newer TIEs received in TIDEs coming from the north. Those headers are then propagated southbound towards the leaf to cause it to originate a higher sequence number of the TIE, effectively refreshing it all the way up to ToF. 🚀
6.4. Reachability Computation 🗺️
A node has three possible sources of relevant information for reachability computation. 🧩 A node knows the full topology south of it from the received North Node TIEs, or alternately north of it from the South Node TIEs. A node has the set of prefixes with their associated distances and bandwidths from corresponding prefix TIEs.
To compute prefix reachability, a node conceptually runs a northbound and a southbound SPF. Here, N-SPF and S-SPF notation denotes the direction in which the computation front is progressing. 🧭
Since neither computation can "loop", it is possible to compute non-equal costs or even k-shortest paths [EPPSTEIN] and "saturate" the fabric to the extent desired. 💪 This specification, however, uses simple, familiar SPF algorithms and concepts as examples, due to their prevalence in today's routing. 📚
For reachability computation purposes, RIFT considers all parallel links between two nodes to be of the same cost, advertised in the _cost_ element of _NodeNeighborsTIEElement_. In case the neighbor has multiple parallel links at different costs, the largest distance (highest numerical value) MUST be advertised. Given the range of Thrift encodings, _infinite_distance_ is defined as the largest non-negative _MetricType_. Any link with a metric larger than that (i.e., the negative MetricType) MUST be ignored in computations. Any link with the metric set to _invalid_distance_ MUST also be ignored in computation. In case of a negatively distributed prefix, the metric attribute MUST be set to _infinite_distance_ by the originator, and it MUST be ignored by all nodes during computation, except for the purpose of determining transitive propagation and building the corresponding routing table. 🌈
A prefix can carry the _directly_attached_ attribute to indicate that the prefix is directly attached, i.e., should be routed to even if the node is in overload. 🛣️ In case of a negatively distributed prefix, this attribute MUST NOT be included by the originator, and it MUST be ignored by all nodes during SPF computation. If a prefix is locally originated, the attribute _from_link_ can indicate the interface to which the address belongs. In case of a negatively distributed prefix, this attribute MUST NOT be included by the originator, and it MUST be ignored by all nodes during computation. A prefix can also carry the _loopback_ attribute to indicate the said property. 🔁
Prefixes are carried in different types of TIEs, indicating their type. For the same prefix being included in different TIE types, tie-breaking is performed according to Section 6.8.1. If the same prefix is included multiple times in multiple TIEs of the same type originating at the same node, the resulting behavior is unspecified. 🤷♀️
6.4.1. Northbound Reachability SPF ⬆️
N-SPF MUST use exclusively northbound and East-West adjacencies in the computing node's North Node TIEs (since if the node is a leaf, it may not have generated a South Node TIE) when starting SPF. 🌱 Observe that N-SPF is really just a one-hop variety, since South Node TIEs are not reflooded southbound beyond a single level (or East-West). So the computation cannot progress beyond adjacent nodes. 🛑
Once progressing, the computation uses the next higher level's South Node TIEs to find corresponding adjacencies to verify backlink connectivity. Two unidirectional links MUST be associated to confirm bidirectional connectivity, a process often known as "backlink check". 🔗 As part of the check, both Node TIEs MUST contain the correct System IDs and expected levels. 🪪
The default route found when crossing an E-W link SHOULD be used if and only if
- the node itself does not have any northbound adjacencies and
- the adjacent node has one or more northbound adjacencies.
This rule forms a "one-hop default route split-horizon". It prevents looping over default routes while allowing for "one-hop protection" of nodes that lost all northbound adjacencies, except at the ToF, where the links are used exclusively to flood topology information in multi-plane designs. 🛡️
Other south prefixes found when crossing E-W links MAY be used if and only if
- no north neighbors are advertising the same or a supersuming non-default prefix and
- the node does not originate a non-default supersuming prefix itself.
That is, the E-W link can be used as a gateway of last resort for a specific prefix only. 🚪 Using south prefixes across an E-W link can be beneficial, e.g., on automatic disaggregation in pathological fabric partitioning scenarios. 🩹
A detailed example can be found in Appendix B.4. 📖
6.4.2. Southbound Reachability SPF ⬇️
S-SPF MUST use the southbound adjacencies in the South Node TIEs exclusively, i.e., progresses towards nodes at lower levels. Observe that E-W adjacencies are NEVER used in this computation. 🙅♀️ This enforces the requirement that a packet traversing in a southbound direction must never change its direction. 🧱
S-SPF MUST use northbound adjacencies in North Node TIEs to verify backlink connectivity, by checking for the presence of the link beside the correct System ID and level. 🔍
6.4.3. East-West Forwarding Within a Non-ToF Level ↔️
Using south prefixes over horizontal links MAY occur if the N-SPF includes East-West adjacencies in computation. 🌉 It can protect against pathological fabric partitioning cases that leave only paths to destinations that would necessitate multiple changes of forwarding direction between north and south. 🔄
6.4.4. East-West Links Within a ToF Level 👑
E-W ToF links behave, in terms of flooding scopes defined in Section 6.3.4, like northbound links, and MUST be used exclusively for control plane information flooding. 📡 Even though a ToF node could be tempted to use those links during southbound SPF and carry traffic over them, this MUST NOT be attempted, since it may, in anycast cases, lead to routing loops. 🌀 An implementation MAY try to resolve the looping problem by following on the ring strictly tie-broken shortest-paths only, but the details are outside this specification. And even then, the problem of proper capacity provisioning of such links, when they become traffic-bearing in case of failures, is vexing. 😩 When used for forwarding purposes, they defeat the statistical non-blocking guarantees that Clos is providing normally. 🧮
6.5. Automatic Disaggregation on Link & Node Failures 🌸
6.5.1. Positive, Non-Transitive Disaggregation ✨
Okay bestie, so normally a node's South TIEs are super chill 💅. They just carry the adjacencies and a default route. BUT if a node notices that its default IP prefix covers one or more prefixes it can reach while one or more other nodes at the same level can't, then it MUST explicitly advertise those prefixes in a South TIE 🚨. Otherwise some percentage of the northbound traffic for those prefixes gets sent to nodes with no reachability and just gets dropped 💀. Even when nothing is dropped, the packets can "backhaul" up through the higher-level spines, which is clearly not it and messes with the blocking probabilities of the fabric 🙈.
The spec calls advertising extra prefixes southbound "positive disaggregation" 🎀. It's non-transitive, so its effects always stay inside a single level of the fabric. Multiple node or link failures can lead to several separate instances of positive disaggregation, and those are needed to prevent looping or bow-tying the fabric 🧶.
A node figures out which prefixes need disaggregating with these steps:
- A DAG computation in the southern direction goes first. The North TIEs are used to find all of the prefixes the node can reach and the set of next hops in the lower level for each of them. This is easy on a fat tree: set all link costs in the southern direction to 1 and all northern directions to infinity. The set of those prefixes is
|R; for each prefix r in|R, its set of next hops is|H(r). - The node uses reflected South TIEs to find all nodes at the same level in the same PoD and the set of southbound adjacencies for each. The set of nodes at the same level is
|N, and for each node n in|N, its set of southbound adjacencies is|A(n). - For a given r, if the intersection of
|H(r)and|A(n), for any n, is empty, then that prefix r must be explicitly advertised by the node in a South TIE. - An identical set of disaggregated prefixes is flooded on each of the node's southbound adjacencies. Following the normal flooding rules for a South TIE, a node at the lower level that receives this South TIE SHOULD NOT propagate it southbound or reflect the disaggregated prefixes back over its adjacencies to nodes at the level it came from.
To sum it all up in the simplest terms: if a node sees that its default route covers prefixes for which one of the other nodes in its level has no possible next hops in the level below, it has to disaggregate them to prevent traffic loss or suboptimal routing through those nodes 🧠. So a node X needs to check whether it can reach a different set of south neighbors than other nodes at the same level, where both are connected through at least one common south neighbor. If it can, prefix disaggregation may be required. If it can't, no prefix disaggregation is needed. There's an example of disaggregation in Appendix B.3 if you want to see it in action 🦋.
Here's a possible algorithm, finally:
- Create
partial_neighbors = (empty), a set of neighbors with partial connectivity to the node X's level from X's perspective. Each entry in the set is a south neighbor of X and a list of nodes of X.level that can't reach that neighbor. - A node X determines its set of southbound neighbors
X.south_neighbors. - For each South TIE originated from a node Y that X has, which is at X.level, if
Y.south_neighborsis not the same asX.south_neighborsbut the nodes share at least one southern neighbor, then for each neighbor N inX.south_neighborsbut not inY.south_neighbors, add(N, (Y))topartial_neighborsif N isn't there, or add Y to the list for N. - If
partial_neighborsis empty, then node X does not disaggregate any prefixes. If node X is advertising disaggregated prefixes in its South TIE, X SHOULD remove them and re-advertise its South TIEs.
A node X computes reachability to all nodes below it based on the received North TIEs first. This gives a set of routes, each categorized by (prefix, path_distance, next-hop set). For clarity in the following procedure, these can also be organized by next-hop set as ((next-hops), {(prefix, path_distance)}). If partial_neighbors isn't empty, then the procedure in Figure 17 describes how to identify the prefixes to disaggregate 🔍.
disaggregated_prefixes = { empty }
nodes_same_level = { empty }
for each South TIE
if (South TIE.level == X.level and
X shares at least one S-neighbor with X)
add South TIE.originator to nodes_same_level
end if
end for
for each next-hop-set NHS
isolated_nodes = nodes_same_level
for each NH in NHS
if NH in partial_neighbors
isolated_nodes =
intersection(isolated_nodes,
partial_neighbors[NH].nodes)
end if
end for
if isolated_nodes is not empty
for each prefix using NHS
add (prefix, distance) to disaggregated_prefixes
end for
end if
end for
copy disaggregated_prefixes to X's South TIE
if X's South TIE is different
schedule South TIE for flooding
end if
Figure 17: Computation of Disaggregated PrefixesEach disaggregated prefix is sent with its corresponding path_distance 📏. This lets a node send the same South TIE to every south neighbor, and the south neighbor that's connected to that prefix will get a shorter path 🌈.
Last but not least, here are the sneaky details the algorithms left out so they'd stay easier to read 👀:
- All neighbor relationships MUST perform backlink checks.
- The overload flag, introduced in Section 6.8.2 and carried in the _overload_ schema element, has to be respected during the computation. Nodes advertising themselves as overloaded MUST NOT be transited in reachability computation, but they MUST be used as terminal nodes with the prefixes they advertise being reachable.
- All the lower-level nodes are flooded the same disaggregated prefixes, since RIFT doesn't build a South TIE per node, which would complicate things unnecessarily. The lower-level node that can compute a southbound route to the prefix will prefer it over the disaggregated route anyway, based on route preference rules.
- Positively disaggregated prefixes do not have to propagate to lower levels. That way, the disturbance in terms of new flooding is contained to the single level experiencing failures 🫶.
- Disaggregated South Prefix TIEs are not "reflected" by the lower level. Nodes within the same level do not need to be aware of which node computed the need for disaggregation.
- The fabric still supports maximum load balancing properties while not trying to send traffic northbound unless necessary 💖.
When positive disaggregation is triggered, the nodes may issue the necessary disaggregated prefixes at different points in time, because the algorithm is very stable but unsynchronized. For a short while, this can cause an "incast" behavior where the first advertising router, based on the nature of the longest prefix match, attracts all of the traffic 😬. Different implementation strategies can lessen that effect, but those are outside the scope of this specification.
It's worth noticing that, in a single-plane ToF, this disaggregation prevents traffic loss up to (K_LEAF * P) link failures in terms of Section 5.2. Or, to put it another way, it takes at least that many link failures to partition the ToF into multiple planes 👑.
6.5.2. Negative, Transitive Disaggregation for Fallen Leaves 🍂
As explained in Section 5.3, failures in a multi-plane ToF, or more than (K_LEAF * P) links failing in a single-plane design, can generate fallen leaves. This scenario can't be addressed by positive disaggregation alone, so it needs a further mechanism 😤.
6.5.2.1. Cabling of Multiple ToF Planes 🧵
Going back to designs with multiple planes, as shown originally in Figure 3, Figure 18 highlights how the ToF is cabled when there are two planes. It uses dual-rings to distribute all the North TIEs within both planes 🦋.
(Artwork only available as SVG: see
https://www.rfc-editor.org/rfc/rfc9692.html)
Figure 18: Topologically Connected PlanesSection 5.3 already describes how failures in multi-plane fabrics can lead to traffic loss that normal positive disaggregation can't fix. RIFT's negative, transitive disaggregation mechanism is the solution for that, and the next section explains how those mechanisms work in more detail 🔧.
6.5.2.2. Transitive Advertisement of Negative Disaggregates 📣
A ToF node that discovers it cannot reach a fallen leaf SHOULD disaggregate all of the prefixes of that leaf. To do that, it uses negative South Prefix TIEs, which are flooded southwards as usual with the scope defined in Section 6.3.4 🌊.
Transitively, a node explicitly loses connectivity to a prefix when none of its children advertises it and the prefix is negatively disaggregated by all of its parents. When that happens, the node originates the negative prefix further down south. Since the mechanism applies recursively south, the negative prefix may propagate transitively all the way down to the leaf 🪜. This matters because leaves connected to multiple planes by disjoint paths may have to choose the correct plane at the very bottom of the fabric. That way they don't send traffic toward another leaf through a plane where it has "fallen", which would make the traffic loss unavoidable 😭.
When connectivity is restored, a node that disaggregated a prefix withdraws the negative disaggregation through the usual mechanism of re-advertising its TIEs without the negative prefix 🌸.
6.5.2.3. Computation of Negative Disaggregates ✨
Okay so negative prefixes can actually get advertised for two different reasons 🙈 We'll go through them one after the other, bestie.
The first reason is a big reachability check 💅 The node looks at all the North Node TIEs and builds the set of every reachable node using reachability computation over the complete graph, horizontal ToF links included. The node itself is the root of this computation. Then it compares that result with the normal southbound SPF from Section 6.4.2. Whatever shows up as different is the fallen leaves, and all of their attached prefixes get advertised as negative prefixes southbound, but only if the node doesn't consider the prefix reachable in the southbound SPF 😭
The second reason is all about how negative prefixes get used in the computation (Figure 19 is the receipt for this 🧾). When negative prefixes get attached at a certain point in time, the negative prefix might end up with every viable next hop from the shorter-match next hop pruned away. Like, all of its northbound neighbors sent a negative prefix advertisement. That's the trigger for advertising this negative prefix transitively south 🦋 It usually happens because the node is in a plane where the prefix belongs to a fabric leaf that has "fallen" in that plane. And obviously, if one of the northbound switches withdraws its negative advertisement, the node has to withdraw its transitively provided negative prefix too 🫶
6.6. Attaching Prefixes 🎀
After an SPF runs, the resulting reachability info needs to get attached in the form of prefixes. For S-SPF, prefixes from a North TIE get attached to the originating node with that node's next-hop set, and a distance equal to the prefix's cost plus the node's minimized path distance. The RIFT route database is a set of (prefix, prefix-type, attributes, path_distance, next-hop set), and it collects all of these results 📚
N-SPF prefixes from each South TIE also need to be added to the RIFT route database. The N-SPF is really just a stub, so the computing node only has to figure out, for each prefix in a South TIE that came from an adjacent node, which next hops to use to reach that node. Since there may be parallel links, the next hops to use can be a set. The presence of the computing node in the associated South Node TIE is enough to verify that at least one link has bidirectional connectivity. The set of minimum cost next hops from the computing node X to the originating adjacent node is determined from there 🌸
Each prefix has its cost adjusted before being added into the RIFT route database. The cost of the prefix is set to the cost received plus the cost of the minimum distance next hop to that neighbor, while considering its attributes such as mobility per Section 6.8.4. Then each prefix can be added into the RIFT route database with the next-hop set. Ties are broken based upon type first, then distance, and further on PrefixAttributes. Only the best combination is used for forwarding. RIFT route preferences are normalized by the enum RouteType in the Thrift [thrift] model given in Section 7 👑
An example implementation for node X follows:
for each South TIE
if South TIE.level > X.level
next_hop_set = set of minimum cost links to the
South TIE.originator
next_hop_cost = minimum cost link to
South TIE.originator
end if
for each prefix P in the South TIE
P.cost = P.cost + next_hop_cost
if P not in route_database:
add (P, P.cost, P.type,
P.attributes, next_hop_set) to route_database
end if
if (P in route_database):
if route_database[P].cost > P.cost or
route_database[P].type > P.type:
update route_database[P] with (P, P.type, P.cost,
P.attributes,
next_hop_set)
else if route_database[P].cost == P.cost and
route_database[P].type == P.type:
update route_database[P] with (P, P.type,
P.cost, P.attributes,
merge(next_hop_set, route_database[P].next_hop_set))
else
// Not preferred route so ignore
end if
end if
end for
end for
Figure 19: Adding Routes from South TIE Positive and Negative
PrefixesAfter the positive prefixes are attached and tie-broken, negative prefixes get attached and used in case of northbound computation, ideally from the shortest length to the longest. The next-hop adjacencies for a negative prefix are inherited from the longest positive prefix that aggregates it. After that, adjacencies to nodes that advertised negative disaggregation for this prefix get removed 🧹
The rule of inheritance MUST be maintained when the next-hop list for a prefix is modified, because the modification may affect the entries for matching negative prefixes of immediate longer prefix length. For instance, if a next hop is added, then by inheritance it must be added to all the negative routes of immediate longer prefix length, unless it is pruned due to a negative advertisement for the same next hop. Similarly, if a next hop is deleted for a given prefix, then it is deleted for all the immediately aggregated negative routes. This recurses in the case of nested negative prefix aggregations 🔁
The rule of inheritance MUST also be maintained when a new prefix of intermediate length is inserted, or when the immediately aggregating prefix is deleted from the routing table. That makes an even shorter aggregating prefix the one from which the negative routes now inherit their adjacencies. As the aggregating prefix changes, all the negative routes MUST be recomputed, and then again, the process may recurse in case of nested negative prefix aggregations 💀
Although these operations can be computationally expensive, the overall load on devices in the network is low, because these computations are not run very often. Positive route advertisements are always preferred over negative ones. This prevents recursion in most cases, because positive reachability information never inherits next hops 🌈
To make the negative disaggregation less abstract, let's look at an example ToP node, T1, with 4 ToF parents S1..S4 as represented in Figure 20:
+----+ +----+ +----+ +----+ N
| S1 | | S2 | | S3 | | S4 | ^
+----+ +----+ +----+ +----+ W< + >E
| | | | v
|+--------+ | | S
||+-----------------+ |
|||+----------------+---------+
||||
+----+
| T1 |
+----+
Figure 20: A ToP Node with 4 ParentsIf all ToF nodes can reach all the prefixes in the network, with RIFT, they will normally advertise a default route south. An abstract Routing Information Base (RIB), more commonly known as a routing table, stores all types of maintained routes, including the negative ones and "tie-breaks" for the best one. An abstract forwarding table (FIB), on the other hand, retains only the ultimately computed "positive" routing instructions. In T1, those tables would look as illustrated in Figure 21:
+---------+
| Default |
+---------+
|
| +--------+
+---> | via S1 |
| +--------+
|
| +--------+
+---> | via S2 |
| +--------+
|
| +--------+
+---> | via S3 |
| +--------+
|
| +--------+
+---> | via S4 |
+--------+
Figure 21: Abstract RIBIn case T1 receives a negative advertisement for prefix 2001:db8::/32 from S1, a negative route is stored in the RIB (indicated by a "~" sign), while the more specific routes to the complementing ToF nodes are installed in FIB. RIB and FIB in T1 now look as illustrated in Figures 22 and 23, respectively:
+---------+ +-----------------+
| Default | <-------------- | ~2001:db8::/32 |
+---------+ +-----------------+
| |
| +--------+ | +--------+
+---> | via S1 | +---> | via S1 |
| +--------+ +--------+
|
| +--------+
+---> | via S2 |
| +--------+
|
| +--------+
+---> | via S3 |
| +--------+
|
| +--------+
+---> | via S4 |
+--------+
Figure 22: Abstract RIB After Negative 2001:db8::/32 from S1The negative 2001:db8::/32 prefix entry inherits from ::/0, so the positive, more specific routes are the complements to S1 in the set of next hops for the default route. That entry is composed of S2, S3, and S4. In other words, it uses all entries in the default route with a "hole punched" for S1 into them. These are the next hops that are still available to reach 2001:db8::/32, now that S1 advertised that it will not forward 2001:db8::/32 anymore. Ultimately, those resulting next hops are installed in FIB for the more specific route to 2001:db8::/32, as illustrated below 🕳️✨
+---------+ +---------------+
| Default | | 2001:db8::/32 |
+---------+ +---------------+
| |
| +--------+ |
+---> | via S1 | |
| +--------+ |
| |
| +--------+ | +--------+
+---> | via S2 | +---> | via S2 |
| +--------+ | +--------+
| |
| +--------+ | +--------+
+---> | via S3 | +---> | via S3 |
| +--------+ | +--------+
| |
| +--------+ | +--------+
+---> | via S4 | +---> | via S4 |
+--------+ +--------+
Figure 23: Abstract FIB After Negative 2001:db8::/32 from S1Okay bestie, so T1 gets a negative advertisement for 2001:db8:1::/48 from S2, and it gets stored in the RIB again 🦋 Here's what the RIB looks like after the update, in Figure 24:
+---------+ +----------------+ +------------------+
| Default | <----- | ~2001:db8::/32 | <------ | ~2001:db8:1::/48 |
+---------+ +----------------+ +------------------+
| | |
| +--------+ | +--------+ |
+---> | via S1 | +---> | via S1 | |
| +--------+ +--------+ |
| |
| +--------+ | +--------+
+---> | via S2 | +---> | via S2 |
| +--------+ +--------+
|
| +--------+
+---> | via S3 |
| +--------+
|
| +--------+
+---> | via S4 |
+--------+Figure 24: Abstract RIB After Negative 2001:db8:1::/48 from S2
So the negative 2001:db8:1::/48 now inherits from 2001:db8::/32 🌸 That means the positive, more specific routes are the complements to S2 in the set of next hops for 2001:db8::/32, which is S3 and S4. In other words, all the parent's entries get their negative holes punched in again 💅 The FIB in T1 now looks like Figure 25:
+---------+ +---------------+ +-----------------+
| Default | | 2001:db8::/32 | | 2001:db8:1::/48 |
+---------+ +---------------+ +-----------------+
| | |
| +--------+ | |
+---> | via S1 | | |
| +--------+ | |
| | |
| +--------+ | +--------+ |
+---> | via S2 | +---> | via S2 | |
| +--------+ | +--------+ |
| | |
| +--------+ | +--------+ | +--------+
+---> | via S3 | +---> | via S3 | +---> | via S3 |
| +--------+ | +--------+ | +--------+
| | |
| +--------+ | +--------+ | +--------+
+---> | via S4 | +---> | via S4 | +---> | via S4 |
+--------+ +--------+ +--------+Figure 25: Abstract FIB After Negative 2001:db8:1::/48 from S2
Next, S3 stops advertising its service as a default gateway 😭 The entry gets removed from the RIB as usual. To update the FIB, T1 has to eliminate the FIB entry for the default route, plus all the FIB entries that were created for negative routes pointing to the RIB entry being removed (::/0). This happens recursively, first for 2001:db8::/32 and then for 2001:db8:1::/48. The related FIB entries via S3 get removed, as shown in Figure 26:
+---------+ +---------------+ +-----------------+
| Default | | 2001:db8::/32 | | 2001:db8:1::/48 |
+---------+ +---------------+ +-----------------+
| | |
| +--------+ | |
+---> | via S1 | | |
| +--------+ | |
| | |
| +--------+ | +--------+ |
+---> | via S2 | +---> | via S2 | |
| +--------+ | +--------+ |
| | |
| | |
| | |
| | |
| | |
| +--------+ | +--------+ | +--------+
+---> | via S4 | +---> | via S4 | +---> | via S4 |
+--------+ +--------+ +--------+Figure 26: Abstract FIB After Loss of S3
Now say S4 also disaggregates prefix 2001:db8:1::/48 at that same time 🙈 Then the FIB entry for 2001:db8:1::/48 becomes a discard route, and that's the signal for T1 to disaggregate 2001:db8:1::/48 negatively, in a transitive fashion, together with its own children.
Finally, S3 becomes available again as a default gateway, and a negative advertisement comes in from S4 about prefix 2001:db8:2::/48 instead of 2001:db8:1::/48 ✨ Again, a negative route gets stored in the RIB, and the more specific route to the complementing ToF nodes gets installed in the FIB. Since 2001:db8:2::/48 inherits from 2001:db8::/32, the positive FIB routes are chosen by removing S4 from S2, S3, S4. The abstract FIB in T1 now looks like Figure 27:
+-----------------+
| 2001:db8:2::/48 |
+-----------------+
|
+---------+ +---------------+ +-----------------+
| Default | | 2001:db8::/32 | | 2001:db8:1::/48 |
+---------+ +---------------+ +-----------------+
| | | |
| +--------+ | | | +--------+
+---> | via S1 | | | +---> | via S2 |
| +--------+ | | | +--------+
| | | |
| +--------+ | +--------+ | | +--------+
+---> | via S2 | +---> | via S2 | | +---> | via S3 |
| +--------+ | +--------+ | +--------+
| | |
| +--------+ | +--------+ | +--------+
+---> | via S3 | +---> | via S3 | +---> | via S3 |
| +--------+ | +--------+ | +--------+
| | |
| +--------+ | +--------+ | +--------+
+---> | via S4 | +---> | via S4 | +---> | via S4 |
+--------+ +--------+ +--------+Figure 27: Abstract FIB After Negative 2001:db8:2::/48 from S4
6.7. Optional Zero Touch Provisioning (RIFT ZTP) 🧚
Each RIFT node can run in Zero Touch Provisioning (ZTP) mode, which means it has no RIFT-specific configuration 🍓 The exceptions are a ToF, or a node that's explicitly configured to act as a leaf and/or support L2L procedures in the overall topology. A ZTP node fully automatically derives the RIFT parameters it needs by itself once it's attached to the topology. Manually configured nodes and nodes running RIFT ZTP can be mixed freely, and they'll form a valid topology if one is achievable 💜
The level of each node is derived from the offers it receives from its neighbors. Each node (except possibly nodes configured as leaves) tries to attach at the highest possible point in the fabric, and this is the part I'm obsessed with 😍 It guarantees that even if the diffusion front of offers reaches a node from "below" faster than from "above", the node will greedily abandon an already negotiated level derived from nodes topologically below it and properly peer with nodes above 👑
The fabric is very consciously numbered from the top down. This lets PoDs have different heights and keeps the number of configurations needed super low. In fact, the only configuration required is a TOP_OF_FABRIC flag on every node at the top of the fabric 🌈
This section describes the necessary concepts and procedures of the RIFT ZTP operation 🎀
6.7.1. Terminology 💅
okay bestie, the flags and the configured level are lowkey tangled together 🧶 it may take a few reads of this glossary before it clicks, and that's totally normal 🫶
Automatic Level Derivation: Procedures that let nodes with no configured level work out their own level automatically 🧚 This only kicks in if CONFIGURED_LEVEL is undefined ✨
UNDEFINED_LEVEL: A "null" value that means the level hasn't been figured out and hasn't been configured either 🙈 Schemas usually show that as a missing optional value with no defined default.
LEAF_ONLY: An optional config flag you can set on a node so it never leaves the "bottom of the hierarchy" 🌸 The TOP_OF_FABRIC flag and CONFIGURED_LEVEL can't be defined at the same time as this one. It implies a CONFIGURED_LEVEL value of _leaf_level_, and it shows up in the _leaf_only_ schema element.
TOP_OF_FABRIC: A config flag that MUST be provided on all ToF nodes 👑 LEAF_FLAG and CONFIGURED_LEVEL can't be defined at the same time as this flag, and it implies a CONFIGURED_LEVEL value. Honestly it's basically a shortcut for setting the same level on every ToF node, and that's unavoidable because other ZTP nodes need an initial "seed" to figure out their level in the topology 🌱 This flag is a big deal in fabrics with multiple planes, since it's what makes negative disaggregation work (Section 6.5.2) 💖 It's carried in the _top_of_fabric_ schema element. A standards-conforming RIFT implementation implies a CONFIGURED_LEVEL value of _top_of_fabric_level_ when TOP_OF_FABRIC is set. That value is kept reasonably low so ZTP can reconverge fast when things fail ⚡
CONFIGURED_LEVEL: A level value that's provided manually 🎀 When it's defined (so it isn't an UNDEFINED_LEVEL), the node sits out of ZTP and doesn't derive its own level from other nodes' info. The TOP_OF_FABRIC flag gets ignored once this value is defined. LEAF_ONLY can only be set if this value is undefined or set to _leaf_level_ 🍓
DERIVED_LEVEL: The level value that gets computed through automatic level derivation when CONFIGURED_LEVEL is equal to UNDEFINED_LEVEL 🦋
LEAF_2_LEAF: An optional flag you can set on a node so it supports the procedures in Section 6.8.9 💜 It's a capability that implies LEAF_ONLY along with its restrictions. The TOP_OF_FABRIC flag gets ignored if it's set at the same time as this one. It's carried in the _leaf_only_and_leaf_2_leaf_procedures_ schema flag.
LEVEL_VALUE: With ZTP, the original definition of "level" from Section 3.1 gets both stretched and loosened 🌈 First, the level is now called LEVEL_VALUE, and it's the first defined value out of CONFIGURED_LEVEL followed by DERIVED_LEVEL. Second, nodes can now be more than one level apart and still form adjacencies, as long as at least one of them is LEAF_ONLY 😭
Valid Offered Level (VOL): A neighbor's level that arrived in a valid LIE (one that passes every check for adjacency formation while ignoring all clauses about level values) and stays put for the whole holdtime interval on that LIE ✨ Heads up: offers from nodes offering the level value _leaf_level_ do NOT count as VOLs, because no valid DERIVED_LEVEL can come from those. That means the _not_a_ztp_offer_ flag MUST be ignored there 🙈 Offers from LIEs where _not_a_ztp_offer_ is true aren't VOLs either. If a node has parallel adjacencies to the same neighbor, the VOL on each one counts as equivalent. In other words, the newest VOL from any of those adjacencies updates the VOL received from that same node 🫶
Highest Available Level (HAL): The highest defined level value out of all the VOLs received 🔥
Highest Available Level Systems (HALS): The set of nodes offering HAL VOLs 👑
Highest Adjacency ThreeWay (HAT): The highest neighbor level across all the formed _ThreeWay_ adjacencies for the node 🌸
6.7.2. Automatic System ID Selection 🧚
RIFT nodes need a 64-bit System ID that SHOULD be derived as EUI-64 MAC Address Block Large (MA-L) according to [EUI64] 🪪 The organizationally governed part of that ID (24 bits) can be used to make multiple IDs if you need more than one RIFT instance 💅
Now the operational vibe check 🚨 The router MUST make sure this identifier doesn't change super often (or at least doesn't change without first sending all of its TIEs with pretty short lifetimes, so it purges them) 😭 Otherwise the network could end up holding a huge pile of stale TIEs in other nodes. That's not necessarily a disaster, though, if the procedures in Section 9 are implemented 🫧
6.7.3. Generic Fabric Example 🦋
ZTP really makes you think about a fabric that's cabled wrong or in a weird way, and how that topology can still be forced into the "lattice" structure a fabric is supposed to have (plus some extra restrictions) 🕸️ The necessary and sufficient physical cabling is in Figure 28 below. The assumption here is that all nodes live in the same PoD 🏠
+---+
| A | s = TOP_OF_FABRIC
| s | L = LEAF_ONLY
++-++ L2L = LEAF_2_LEAF
| |
+--+ +--+
| |
+--++ ++--+
| E | | F |
| +-+ | +-----------+
++--+ | ++-++ |
| | | | |
| +-------+ | |
| | | | |
| | +----+ | |
| | | | |
++-++ ++-++ |
| I +-----+ J | |
| | | +-+ |
++-++ +--++ | |
| | | | |
+---------+ | +------+ |
| | | | |
+-----------------+ | |
| | | | |
++-++ ++-++ |
| X +-----+ Y +-+
|L2L| | L |
+---+ +---+
Figure 28: Generic ZTP Cabling ConsiderationsFirst, RIFT has to anchor the "top" of the cabling, and that's exactly what the TOP_OF_FABRIC flag on node A is for 👑 Then things look smooth until the protocol has to decide whether node Y sits at the same level as I and J (which would make X south of it), or whether it's X 🤔 That's unresolvable here until we "nail down the bottom" of the topology 🧱 To do that, the protocol uses the leaf flags on X and Y in this example. If Y doesn't have a leaf flag, it'll try to elect the highest level offered and end up at the same level as I and J 🌈
6.7.4. Level Determination Procedure 💜
A node that starts up with UNDEFINED_VALUE (so no CONFIGURED_LEVEL and no leaf or TOP_OF_FABRIC flag) MUST follow these additional procedures 🫶
- It advertises its LEVEL_VALUE on all LIEs. Observe that this can be UNDEFINED_LEVEL, which in schema terms is just an omitted optional value 🙈
- It computes HAL as the numerically highest available level across all VOLs 🔥
- Then it picks MAX(HAL-1,0) as its DERIVED_LEVEL 🦋 The node then starts advertising this derived level ✨
- A node that lost all its adjacencies with the HAL value MUST holddown the computation of the new DERIVED_LEVEL for at least one second, unless it has no VOLs from southbound adjacencies ⏳ Once the holddown timer expires, it MUST discard all received offers, recompute DERIVED_LEVEL, and announce it to all neighbors 📣
- A node MUST reset any adjacency that changed the level it's offering and is in
_ThreeWay_state 💅 - A node that changed its defined level value MUST re-advertise its own TIEs, since the new
_PacketHeader_will carry a different level than before 🔁 The sequence number of each TIE MUST be increased 🔢 - After a level has been derived, the node MUST set
_not_a_ztp_offer_on LIEs towards all systems offering a VOL for HAL 🚫 - A node that changed its level SHOULD flush the TIEs of all other nodes from its LSDB 🧹 Otherwise stale info can stick around on "direction reversal", meaning nodes that used to seem south are now north or east-west 🙃 This won't stop the protocol from working correctly, but it could be a little confusing operationally 😵💫
A node that starts with LEVEL_VALUE being 0 (so it assumes a leaf function by being configured with the right flags, or it has a CONFIGURED_LEVEL of 0) MUST follow this additional procedure 🌱
- It computes HAT per the procedures above, but does not use it to compute DERIVED_LEVEL. HAT is used to limit adjacency formation per Section 6.2 🔒
It MAY also follow this modified procedure 🎀
- It may pick a different strategy to choose VOL, e.g., use the VOL value with the highest number of VOLs 🧠 Strategies like this are only possible because the node always stays "at the bottom of the fabric". Meanwhile another layer could "invert" the fabric by picking its preferred VOL in a different way, instead of always chasing the highest viable level 🙃✨
6.7.5. RIFT ZTP FSM 🎀
okay so this section is the precise, normative ZTP FSM, and you can skip it unless you're actually building an implementation 🤓 Figure 29 shows the same thing visually, and if you want the formal schema, check Section 7 ✨
The initial state is ComputeBestOffer 🌸
Enter
|
v
+------------------+
| ComputeBestOffer |
| |<----+
| | | ChangeLocalHierarchyIndications
| | | ChangeLocalConfiguredLevel
| | | NeighborOffer
| | | BetterHAL
| | | BetterHAT
| | | LostHAT
| | | ShortTic
| |-----+
| |
| |<---------------------
| |---------------------> (UpdatingClients)
| | ComputationDone
+------------------+
^ |
| | LostHAL
| V
(HoldingDown)
(ComputeBestOffer)
| ^
| | ChangeLocalHierarchyIndications
| | ChangeLocalConfiguredLevel
| | HoldDownExpired
V |
+------------------+
| HoldingDown |
| |<----+
| | | NeighborOffer
| | | BetterHAL
| | | BetterHAT
| | | LostHAL
| | | LostHat
| | | ComputationDone
| | | ShortTic
| |-----+
+------------------+
^
|
(UpdatingClients)
(ComputeBestOffer)
| ^
| | ChangeLocalHierarchyIndications
| | ChangeLocalConfiguredLevel
| | BetterHAL
| | BetterHAT
| | LostHAT
V |
+------------------+
| UpdatingClients |
| |<----+
| | |
| | | NeighborOffer
| | | ShortTic
| |-----+
+------------------+
|
| LostHAL
V
(HoldingDown)
Figure 29: RIFT ZTP FSMHere are the well-known procedures you'll keep seeing 💅
- PUSH Event 📬: queues an event to be executed by the FSM upon exit of this action
- COMPARE_OFFERS ⚖️: checks whether, based on current offers and held last results, the events BetterHAL/LostHAL/BetterHAT/LostHAT are necessary and returns them
- UPDATE_OFFER 🔄: store current offer with adjacency holdtime as lifetime and COMPARE_OFFERS, then PUSH corresponding events
- LEVEL_COMPUTE 🧮: compute best offered or configured level and HAL/HAT, if anything changed, PUSH ComputationDone
- REMOVE_OFFER 🗑️: remove the corresponding offer and COMPARE_OFFERS, PUSH corresponding events
- PURGE_OFFERS 🧹: REMOVE_OFFER for all held offers, COMPARE OFFERS, PUSH corresponding events
- PROCESS_OFFER 🔍:
- if no level is offered, then REMOVE_OFFER
- else
- a. if offered level > leaf, then UPDATE_OFFER
- b. else REMOVE_OFFER
The states are where things live 🏠
- ComputeBestOffer: Processes received offers to derive ZTP variables.
- HoldingDown: Holding down while receiving updates.
- UpdatingClients: Updates other FSMs on the same node with computation results.
Now the events, which is the fun part 🎉
- ChangeLocalHierarchyIndications: Node locally configured with new leaf flags.
- ChangeLocalConfiguredLevel: Node locally configured with a defined level.
- NeighborOffer: A new neighbor offer with optional level and neighbor state.
- BetterHAL: Better HAL computed internally.
- BetterHAT: Better HAT computed internally.
- LostHAL: Lost last HAL in computation.
- LostHAT: Lost HAT in computation.
- ComputationDone: Computation performed.
- HoldDownExpired: Holddown timer expired.
- ShortTic ⏱️: One-second timer tick. This event is provided to the FSM once a second by an implementation-specific mechanism that is outside the scope of this specification. This event is quietly ignored if the relevant transition does not exist.
And the actions, aka what the FSM does when each transition fires 🔥
on ChangeLocalConfiguredLevelin HoldingDown finishes in ComputeBestOffer: store configured levelon BetterHATin HoldingDown finishes in HoldingDown: no actionon ShortTicin HoldingDown finishes in HoldingDown: remove expired offers, and if holddown timer expired, PUSH_EVENT HoldDownExpiredon NeighborOfferin HoldingDown finishes in HoldingDown: PROCESS_OFFERon ComputationDonein HoldingDown finishes in HoldingDown: no actionon BetterHALin HoldingDown finishes in HoldingDown: no actionon LostHATin HoldingDown finishes in HoldingDown: no actionon LostHALin HoldingDown finishes in HoldingDown: no actionon HoldDownExpiredin HoldingDown finishes in ComputeBestOffer: PURGE_OFFERSon ChangeLocalHierarchyIndicationsin HoldingDown finishes in ComputeBestOffer: store leaf flagson LostHATin ComputeBestOffer finishes in ComputeBestOffer: LEVEL_COMPUTEon NeighborOfferin ComputeBestOffer finishes in ComputeBestOffer: PROCESS_OFFERon BetterHATin ComputeBestOffer finishes in ComputeBestOffer: LEVEL_COMPUTEon ChangeLocalHierarchyIndicationsin ComputeBestOffer finishes in ComputeBestOffer: store leaf flags and LEVEL_COMPUTEon LostHALin ComputeBestOffer finishes in HoldingDown: if any southbound adjacencies present, then update holddown timer to normal duration, else fire holddown timer immediatelyon ShortTicin ComputeBestOffer finishes in ComputeBestOffer: remove expired offerson ComputationDonein ComputeBestOffer finishes in UpdatingClients: no actionon ChangeLocalConfiguredLevelin ComputeBestOffer finishes in ComputeBestOffer: store configured level and LEVEL_COMPUTEon BetterHALin ComputeBestOffer finishes in ComputeBestOffer: LEVEL_COMPUTEon ShortTicin UpdatingClients finishes in UpdatingClients: remove expired offerson LostHALin UpdatingClients finishes in HoldingDown: if any southbound adjacencies are present, then update holddown timer to normal duration, else fire holddown timer immediatelyon BetterHATin UpdatingClients finishes in ComputeBestOffer: no actionon BetterHALin UpdatingClients finishes in ComputeBestOffer: no actionon ChangeLocalConfiguredLevelin UpdatingClients finishes in ComputeBestOffer: store configured levelon ChangeLocalHierarchyIndicationsin UpdatingClients finishes in ComputeBestOffer: store leaf flagson NeighborOfferin UpdatingClients finishes in UpdatingClients: PROCESS_OFFERon LostHATin UpdatingClients finishes in ComputeBestOffer: no actionon Entry into ComputeBestOffer: LEVEL_COMPUTEon Entry into UpdatingClients: update all LIE FSMs with computation results
6.7.6. Resulting Topologies 🗺️
the procedures in Section 6.7.4 end up building the RIFT topology and levels shown in Figure 30 💖
+---+
| A |
| 24|
++-++
| |
+--+ +--+
| |
+--++ ++--+
| E | | F |
| 23+-+ | 23+-----------+
++--+ | ++-++ |
| | | | |
| +-------+ | |
| | | | |
| | +----+ | |
| | | | |
++-++ ++-++ |
| I +-----+ J | |
| 22| | 22| |
++--+ +--++ |
| | |
+---------+ | |
| | |
++-++ +---+ |
| X | | Y +-+
| 0 | | 0 |
+---+ +---+
Figure 30: Generic ZTP Topology AutoconfiguredNow here's the plot twist 😭 If the LEAF_ONLY restriction on Y is removed, the outcome is totally different and you get Figure 31. This basically shows that autoconfiguration makes miscabling detection hard 🙈 and, with that, it can lead to undesirable effects when leaves aren't "nailed" by the appropriately configured flags and are cabled arbitrarily. Not me being like "wait, that's a problem" 💀
+---+
| A |
| 24|
++-++
| |
+--+ +--+
| |
+--++ ++--+
| E | | F |
| 23+-+ | 23+-------+
++--+ | ++-++ |
| | | | |
| +-------+ | |
| | | | |
| | +----+ | |
| | | | |
++-++ ++-++ +-+-+
| I +-----+ J +-----+ Y |
| 22| | 22| | 22|
++-++ +--++ ++-++
| | | | |
| +-----------------+ |
| | |
+---------+ | |
| | |
++-++ |
| X +--------+
| 0 |
+---+
Figure 31: Generic ZTP Topology Autoconfigured6.8. Further Mechanisms ✨
6.8.1. Route Preferences ✨
okay so RIFT is SO good at telling route types apart 💅 like external routes from other protocols, plus it also advertises special routes when disaggregation happens. That means the protocol MUST tie-break internally different types on a clear preference scale to prevent traffic loss or loops 🙈 The preferences live in the schema type RouteType.
Table 5 is the cheat sheet 📋 for which route type you get from each TIE type. Entries are sorted from the most preferred route type to the least preferred one.
+==================================+======================+
| TIE Type | Resulting Route Type |
+==================================+======================+
| None | Discard |
+----------------------------------+----------------------+
| Local Interface | LocalPrefix |
+----------------------------------+----------------------+
| S-PGP | South PGP |
+----------------------------------+----------------------+
| N-PGP | North PGP |
+----------------------------------+----------------------+
| North Prefix | NorthPrefix |
+----------------------------------+----------------------+
| North External Prefix | NorthExternalPrefix |
+----------------------------------+----------------------+
| South Prefix and South Positive | SouthPrefix |
| Disaggregation | |
+----------------------------------+----------------------+
| South External Prefix and South | SouthExternalPrefix |
| Positive External Disaggregation | |
+----------------------------------+----------------------+
| South Negative Prefix | NegativeSouthPrefix |
+----------------------------------+----------------------+
Table 5: TIEs and Contained Route Types6.8.2. Overload Bit 🚩
the overload attribute lives in the packet encoding schema (Section 7) as the overload flag, and it is a big deal 😭
The overload flag MUST be respected by all necessary SPF computations. A node with the overload flag set SHOULD advertise all locally hosted prefixes, both northbound and southbound; all other southbound prefixes SHOULD NOT be advertised.
Leaf nodes SHOULD set the overload attribute on all originated Node TIEs. Here's the vibe: if spine nodes forwarded traffic that wasn't meant for the local node, the leaf couldn't stop routing/forwarding loops because it doesn't have the topology info it would need to do that 💀
6.8.3. Optimized Route Computation on Leaves 🍃
leaf nodes only see their directly connected nodes, so they are not required to run "full" SPF computations. Instead, they can gather prefixes from neighboring nodes and run a "partial" SPF computation to build the routing table. Lowkey genius 🧠
Leaf nodes SHOULD only hold their own N-TIEs and, in cases of L2L implementations, the N-TIEs of their East-West neighbors. Leaf nodes MUST hold all S-TIEs from their neighbors.
Normally, a full network graph gets built from local N-TIEs and remote S-TIEs received from neighbors, and then the necessary SPF computations happen. But leaf nodes can skip all that drama 🎀 They can simply compute the minimum cost and next-hop set of each leaf neighbor by checking their local adjacencies. Associated N-TIEs are used to determine bidirectionality and derive the next-hop set. The cost is then derived from the minimum cost of the local adjacency to the neighbor plus the prefix cost.
Leaf nodes would then attach the necessary prefixes as described in Section 6.6. ✨
6.8.4. Mobility 🦋
The RIFT control plane MUST maintain the real time status of every prefix: which port it's attached to, and which leaf node that port belongs to. This is still true for IP mobility, even when the point of attachment may change several times a second 🏃♀️💨
There are two classic approaches to explicitly maintain this info, "timestamp" and "sequence counter". Here's how they work:
timestamp: With this method, the infrastructure SHOULD record the precise time at which the movement is observed. One key advantage is that it has no dependency on the mobile device, which is so clean 🫶 One drawback is that the infrastructure MUST be precisely synchronized so it can compare timestamps as points of attachment change. This could be done with the Precision Time Protocol (PTP) (IEEE Std. 1588 [IEEEstd1588] or 802.1AS [IEEEstd8021AS]), which is designed for bridged LANs. Both the precision of the synchronization protocol and the resolution of the timestamp must beat the shortest possible roaming time on the fabric. Another drawback is that the presence of a mobile device may only be observed asynchronously, such as when it starts using an IP protocol like ARP [RFC0826], IPv6 Neighbor Discovery [RFC4861], IPv6 Stateless Address Configuration [RFC4862], DHCP [RFC2131], or DHCPv6 [RFC8415].
sequence counter: With this method, a mobile device tells its point of attachment on arrival using a sequence counter that gets incremented on each movement. The upside is that it doesn't need a precise sense of time, since the mobile device keeps the movements in order itself 🧚 The downside is the need for support for protocols the mobile device uses to register its presence with the leaf node, with the capability to provide a sequence counter. Well-known issues with sequence counters, such as wrapping and comparison rules, MUST be addressed properly. Sequence numbers MUST be compared by a single homogenous source to make operation feasible. Comparing sequence numbers from multiple heterogeneous sources would be extremely difficult to implement 😭
RIFT supports a hybrid approach using an optional PrefixSequenceType attribute (which is also called a monotonic_clock in the schema). It consists of a timestamp and an optional sequence number field. For a negatively distributed prefix, this attribute MUST NOT be included by the originator, and it MUST be ignored by all nodes during computation. When this attribute is present (observe that, per the data schema, the attribute itself is optional, but if it's included, the timestamp field is required):
- The leaf node MAY advertise a timestamp of the latest sighting of a prefix, e.g., by snooping IP protocols, or the node can use the time at which it advertised the prefix. RIFT transports the timestamp within the desired North Prefix TIEs as the [IEEEstd1588] timestamp.
- RIFT MAY interoperate with "Registration Extensions for 6LoWPAN Neighbor Discovery" [RFC8505], which provides a method for registering a prefix with a sequence number called a Transaction ID (TID). In such cases, RIFT SHOULD transport the derived TID without modification.
- RIFT also defines an abstract negative clock (ASNC) (also called an "undefined" clock). The ASNC MUST be considered older than any other defined clock. By default, when a node receives a North Prefix TIE that does not contain a
PrefixSequenceTypeattribute, it MUST interpret the absence as the ASNC. - Any prefix present on the fabric in multiple nodes that have the same clock is considered anycast.
- The RIFT specification assumes that all nodes are synchronized within at least 200 milliseconds or less. This is achievable through the use of NTP [RFC5905]. An implementation MAY provide a way to reconfigure a domain to a different value, and it provides a variable called
MAXIMUM_CLOCK_DELTAfor this purpose. 🕰️
6.8.4.1. Clock Comparison ⏰
All monotonic clock values MUST be compared to each other using the following rules:
- The ASNC is older than any other value except ASNC and
- Clocks with timestamps differing by more than
MAXIMUM_CLOCK_DELTAare comparable by using the timestamps only and - Clocks with timestamps differing by less than
MAXIMUM_CLOCK_DELTAare comparable by using their TIDs only, and - An undefined TID is always older than any other TID, and
- TIDs are compared using rules of [RFC8505].
6.8.4.2. Interaction Between Timestamps and Sequence Counters 🤝
For attachment changes that happen less frequently (e.g., once per second), the timestamp that the RIFT infrastructure captures should be enough to figure out the most current discovery. If the point of attachment changes faster than the maximum drift of the timestamping mechanism (i.e., MAXIMUM_CLOCK_DELTA), then a sequence number SHOULD be used to get the precision needed to determine currency.
The sequence counter in [RFC8505] is encoded as one octet and wraps around using the arithmetic defined in Appendix A.
Within the resolution of MAXIMUM_CLOCK_DELTA, sequence counter values captured during 2 sequential iterations of the same timestamp SHOULD be comparable. So with default values, a node can move up to 127 times in a 200-millisecond period and the clocks will still be comparable 🤯 This lets the RIFT infrastructure explicitly assert the most up-to-date advertisement. 💖
6.8.4.3. Anycast vs. Unicast 📍
A unicast prefix can be attached to one leaf at most, whereas an anycast prefix may be reachable via more than one leaf.
If a monotonic clock attribute is provided on the prefix, then the prefix with the newest clock value is strictly preferred. An anycast prefix does not carry a clock, or all clock attributes MUST be the same under the rules of Section 6.8.4.1.
During mobility events, it's important that the leaf reflood as quickly as possible to communicate the absence of the prefix that moved 🏎️
Without support for [RFC8505], movements on the fabric within intervals smaller than 100 msec will be interpreted as anycast.
6.8.4.4. Overlays and Signaling 🌈
RIFT is agnostic to any overlay technologies and their associated control and transports that run on top of it (e.g., Virtual eXtensible Local Area Network (VXLAN)). It is expected that leaf nodes and possibly ToF nodes can perform the necessary data plane encapsulation.
In the context of mobility, overlays give another possible solution to avoid injecting mobile prefixes into the fabric, and they can improve the scalability of the deployment too. It makes sense to consider overlays for mobility solutions in IP fabrics. As an example, a mobility protocol such as the Locator/ID Separation Protocol (LISP) [RFC9300] [RFC9301] may inform the ingress leaf of the location of the egress leaf in real time 🛸
Another possibility is to consider mobility as an underlay service and support it in RIFT to an extent. The load on the fabric increases with the amount of mobility, since a move forces flooding and computation on all nodes in the scope of the move. So tunneling from the leaf to the ToF may be desired to speed up convergence times. 🚀
6.8.5. Key/Value (KV) Store 🗄️
6.8.5.1. Southbound 💖
okay so RIFT can totally send key-value pairs southbound 🫶 which is super useful for passing down stuff like configuration info to the levels below. KV South TIEs can show up from a bunch of different nodes, so the nodes MUST run these tie-breaking rules for each key 👑:
- Only KV TIEs from nodes that have a bidirectional adjacency MUST be considered. no adjacency, no vibes 🙅♀️
- For each valid KV South TIE with the same key, the value in the South TIE with the highest level wins 🏆. If the levels are the same, the highest originating System ID wins instead. And if the winning South TIE has overlapping keys, the behavior is undefined 💀 (nobody knows what happens, bestie, so don't count on it 😭)
ngl this part is lowkey so iconic 🦋 because if a node goes down, the nodes south of it lose their adjacencies, which means they stop trusting the KVs they got from it 🧚. New KV South TIEs get advertised so the nodes further south don't end up using stale info 🙈. And the southbound KV advertisements aren't each node doing its own independent math over the same South TIEs. It's a diffused computation 🌸, which is giving distributed teamwork energy 💅
6.8.5.2. Northbound 🌈
Some use cases need really important KV info that the leaves generate, and that info goes northbound 🍓. It gets flooded in KV North TIEs. The originator of a KV North TIE is preserved while it floods, so if the meaning of the info needs tie-breaking rules, the mechanism that defines it will add them 🎀
Only KV TIEs from nodes that are reachable via the multi-plane reachability computation in Section 6.5.2.3 SHOULD be considered ✨
6.8.6. Interactions with BFD 🫧
RIFT MAY use Bidirectional Forwarding Detection (BFD) [RFC5881] to react super fast to link failures ⚡ If it does, these procedures come into play:
- After RIFT _ThreeWay_ hello adjacency convergence, a BFD session MAY form automatically between the RIFT endpoints, no extra config needed 🪄. It uses the exchanged discriminators, which are equal to the _local_id_ in the _LIEPacket_. The remote side's BFD support is carried in the LIEs under _LinkCapabilities_ 💜
- If an established BFD session goes Down after it was Up, the RIFT adjacency SHOULD be re-initialized and then started again from Init once it gets a consecutive BFD Up 🔄
- With parallel links between nodes, each link MAY run its own independent BFD session, or they MAY share one 🤝. How that's actually implemented is outside the scope of this document, so that's a later problem 🙃
- If link identifiers or BFD capabilities change, both the LIE and any BFD sessions SHOULD be brought down and back up again 🔁. But if only the advertised capabilities change, the node MAY choose to keep the BFD session alive 💅
- Multiple RIFT instances MAY share a single BFD session 👯. In that case, which discriminators get used is undefined 💀. RIFT MAY advertise the same link ID for the same interface across multiple instances to "share" discriminators, though 🌸
- The BFD TTL follows [RFC5082] 🎀
6.8.7. Fabric Bandwidth Balancing ⚖️
Okay so everyone knows this one: when links fail in a fabric, it would be ideal to rebalance how much traffic goes to the switches in the next level, based on the ingress and egress bandwidth that's actually available 🥺
RIFT has a light-weight mechanism for this, and it works because RIFT is loop-free 🧚♀️
6.8.7.1. Northbound Direction 🚀
Every RIFT node SHOULD work out how much northbound bandwidth is available through its neighbors at a higher level, then tweak the distance it got on the default route from those neighbors 🛣️. The bandwidth gets advertised in the _NodeNeighborsTIEElement_, which is the sum of the bandwidths of all the parallel links to a neighbor 🫶. Default routes with different distances SHOULD be used for weighted ECMP forwarding 🍰. That distance is called the Bandwidth Adjusted Distance (BAD) 💅. A simple example makes this way clearer, so here it is 👇
100 x 100 100 Mbit/s
| x | |
+-+---+-+ +-+---+-+
| | | |
|Spin111| |Spin112|
+-+---+++ ++----+++
|x || || ||
|| |+---------------+ ||
|| +---------------+| ||
|| || || ||
|| || || ||
-----All Links 10 Mbit/s-----
|| || || ||
|| || || ||
|| +------------+| || ||
|| |+------------+ || ||
|x || || ||
+-+---+++ +--++-+++
| | | |
|Leaf111| |Leaf112|
+-------+ +-------+
Figure 32: Balancing BandwidthFigure 32 shows a topology where the leaf-to-spine links are 10 Mbit/s and the links going north from the spines are 100 Mbit/s 🌈. There's a parallel link failure between Leaf 111 and Spine 111, so Leaf 111 wants to push more traffic toward Spine 112 😤. There's also an uplink failure on Spine 111 💀
The local tweak to the distance the node got on the default route from the upper level happens through a pretty simple algorithm 🧮. The bandwidth gets weighted exponentially, and the distance on the default route acts as a multiplier on that bandwidth weight, which makes operational adjustments easy 🎀
On a node L, use the Node TIEs to compute 3 values from each non-overloaded northbound neighbor N ✨:
- L_N_u: the sum of the bandwidth available from L to N (to account for parallel links) 🔗
- N_u: the sum of the uplink bandwidth available on N ⬆️
- T_N_u: L_N_u * OVERSUBSCRIPTION_CONSTANT + N_u 🧠
For all the T_N_u values, find the matching M_N_u as log_2(next_power_2(T_N_u)), and find MAX_M_N_u as the biggest of all those M_N_u values 📈
For each advertised default route from a node N, change the advertised distance D to BAD = D * (1 + MAX_M_N_u - M_N_u), and use BAD instead of D to balance the weight of default forwarding toward N 🫧
Here's a table to make the example click 🍓. The implicit assumption is that every default route distance is advertised with D=1 and that OVERSUBSCRIPTION_CONSTANT=1 💅
+=========+===========+=======+=======+=====+
| Node | N | T_N_u | M_N_u | BAD |
+=========+===========+=======+=======+=====+
| Leaf111 | Spine 111 | 110 | 7 | 2 |
+---------+-----------+-------+-------+-----+
| Leaf111 | Spine 112 | 220 | 8 | 1 |
+---------+-----------+-------+-------+-----+
| Leaf112 | Spine 111 | 120 | 7 | 2 |
+---------+-----------+-------+-------+-----+
| Leaf112 | Spine 112 | 220 | 8 | 1 |
+---------+-----------+-------+-------+-----+
Table 6: BAD ComputationIf a calculation goes over the range of the type (bandwidth, for example), the result gets set to the highest possible value for that type 📏
BAD SHOULD only be computed for default routes 🫶. A node MAY compute and use BAD for any disaggregated prefixes or other RIFT routes 💜. A node MAY use a different algorithm to weight northbound traffic based on bandwidth 🍰. If it does, its successful behavior MUST NOT depend on the algorithm being uniform, or on the BAD computations being synchronized across the fabric 🚫. For example, leaves could use real-time link loads gathered by analytics to change how much traffic goes to each default route next hop 📊
A change in available bandwidth only affects, at most, two levels down in the fabric 🧚. So the blast radius of bandwidth adjustments stays constrained no matter how tall the fabric is 🎀
6.8.7.2. Southbound Direction 🌸
Because it's loop-free, a node MAY, during South SPF, account for the maximum available bandwidth on nodes in lower levels and change how much traffic it offers to the southbound nodes of the next level 🧠. Such computations might work better if they're standardized, but they don't have to be 🤷♀️. As long as a packet keeps flowing southbound, it'll take some viable, loop-free path to reach its destination 🦋
6.8.8. Label Binding 🏷️
In its LIEs, a node MAY advertise a locally significant, downstream-assigned, interface-specific label 💖. One use for that label is a hop-by-hop encapsulation, which lets forwarding planes be easily told apart when multiple RIFT instances are running 🫶
6.8.9. L2L Procedures 👯♀️
RIFT implementations SHOULD support special East-West adjacencies between leaf nodes ✨. Leaf nodes that support these procedures MUST:
- advertise the LEAF_2_LEAF flag in their node capabilities 🏳️🌈,
- set the overload flag on all of the leaf's Node TIEs 🚩,
- flood only a node's own North and South TIEs over E-W leaf adjacencies 📨,
- always use the E-W leaf adjacency in all SPF computations 🧮,
- install a discard route for any advertised aggregate routes in a leaf's TIE, and 🗑️
- never form southbound adjacencies 🙅♀️.
This lets E-W leaf nodes trade traffic strictly for the prefixes advertised in each other's North Prefix TIEs 💅. The southbound computation finds the reverse direction in the other node's TIE and installs its north prefixes, so it all lines up 🎀
6.8.10. Address Family and Multi-Topology Considerations 🌍
Multi-Topology (MT) [RFC5120] and Multi-Instance (MI) [RFC8202] are concepts that link-state routing protocols use today to support several domains on the same physical topology 🗺️. RIFT supports this by carrying transport ports in the LIE protocol exchanges 🚪. LIEs can be multiplexed by choosing different multicast addresses or ports on the same address 📡
BFD interactions in Section 6.8.6 are implementation-dependent when multiple RIFT instances run on the same link 🫠
6.8.11. One-Hop Healing of Levels with East-West Links 🩹
Based on the rules in Sections 6.4 and 6.3.8, and given the presence of E-W links, RIFT can provide one-hop protection for nodes that have lost all their northbound links 🦸♀️. This can also apply to multi-plane designs where complex link set failures happen at the ToF, when links are used exclusively for flooding topology information 🌩️. Appendix B.4 outlines this behavior 📖✨
6.9. Security 🔐
6.9.1. Security Model ✨
okay so security and ZTP (zero-touch provisioning) are literally always in a tug-of-war 💅 the tighter you lock down who can talk to who, the more the provisioning has to do, and ZTP gets less to do on its own. at the very least, an adjacency that's already up has to stay secure 🔐
RIFT gives operators a few security models to pick from, so they can dial in the vibe they want 🎀
- the most security-conscious operators can control exactly which ports interconnect between a given pair of nodes. this is the "Port-Association Model" (PAM) 💖 you make it happen by configuring each directly connected pair of ports with a designated shared key or a public/private key pair.
- in physically secure data center spots, operators can control connectivity between entire nodes instead. this is the "Node-Association Model" (NAM) 🧚 bonus: it makes port sparing easier ✨
- in the most relaxed environments, an operator only controls which nodes join a particular fabric. this is the "Fabric-Association Model" (FAM) 🦋 it works by using a single shared secret across the entire fabric. it makes sense when servers are treated as leaf devices, since those get swapped out way more often than network nodes. on top of that, this model makes node sparing simpler 🍓
- you can mix these models throughout the fabric, depending on the security needs at different levels and how much extra provisioning complexity an operator is willing to take on 🌈
so, to pull all of this off, RIFT implementations support, under operator control, mechanisms for:
- specifying the appropriate level in the fabric,
- discovering and reporting missing connections, and
- discovering and reporting unexpected connections, while stopping them from forming insecure adjacencies.
here's the thing: operators can choose to configure only the level of each node, and not explicitly configure which connections are allowed. in that case, RIFT only lets adjacencies form between nodes that sit in adjacent levels 🫶 and operators with the lowest security requirements don't have to configure any connection rules at all. nodes in those fabrics can rely fully on ZTP and on adjacencies between nodes in adjacent levels. Figure 33 shows the trade-offs between these models, and it's giving very clear vibes 👑
some level of link quality verification may be required before an adjacency is used for forwarding. for example, an implementation may require that a BFD session comes up before the adjacency is advertised. 🙈
for all the cases above, RIFT has two ways to make sure a local port is connected to the correct port on the correct remote node 💅 the first way is to piggyback on RIFT's authentication mechanism. as long as the provisioning model (for example, YANG) is flexible enough, operators can provision a unique authentication key for:
- each pair of ports in the "port-association model,"
- each pair of switches in the "node-association model," or
- the entire fabric in the "fabric-association model."
the second way is to use the System ID, port-id, and level fields in the LIE message to check an adjacency against the expected cabling topology. optionally, implementations can add some new rules to the FSM so the adjacency comes up when those expectations are met ✨
^ /\ |
/|\ / \ |
| / \ |
| / PAM \ |
Increasing / \ Increasing
Integrity +----------+ Flexibility
& / NAM \ &
Increasing +--------------+ Less
Provisioning / FAM \ Configuration
| / \ |
| +--------------------+ \|/
| / Zero Configuration \ v
+------------------------+
Figure 33: Security Model6.9.2. Security Mechanisms 🔥
okay bestie, the RIFT security goals are to ensure:
- authentication,
- message integrity,
- the prevention of replay attacks,
- low processing overhead, and
- efficient messaging
all of that applies unless no security is deployed, which happens when undefined_securitykey_id is used as the key identifier (key ID) 🙈
message confidentiality is a non-goal, so encryption isn't part of the plan here 💀
the security models from the previous section support a range of security key types, and they line up with the different security association models. PAM and NAM allow security associations at the port or node level, using symmetric or asymmetric keys that are preinstalled 🔑 FAM only applies security associations at a group level, or refines them once the topology has been established 🦋 RIFT doesn't specify how security keys get installed or updated, but it does specify how a key can be used to reach the security goals 🌸
the protocol includes "weak" nonces to help prevent replay attacks, and its authentication mechanisms are comparable to those described in [RFC5709] and [RFC7987] ✨🎀
6.9.3. Security Envelope 💌
Okay bestie, let's talk envelopes 💌 A serialized schema ProtocolPacket MUST be carried in a secure envelope, and Figure 34 shows exactly how that envelope is laid out 🦋 The ProtocolPacket MUST be serialized using the default Thrift's binary protocol, no cap 💅
Any value that comes after a security fingerprint in the packet MUST be used by a receiver only after the fingerprint, generated from an acceptable, advertised key ID, has been validated against the data covered by the bare exceptions arising from operational exigencies 🙈 Depending on local configuration, a node MAY allow the envelope's integrity checks to be skipped, and for the procedure specified in Section 6.9.6 to be implemented 🌸
So for all packets: if the node is set up to validate the outer fingerprint based on a key ID, then an unexpected key ID, or a fingerprint that doesn't validate against the expected key ID, will lead to packet rejection 🚫 And for a TIE: if the receiver is set up to validate the originator by checking the TIE Origin Security Envelope Header fingerprint against a key ID, then an incorrect key ID, or an inner fingerprint that doesn't validate against that key ID, will lead to rejection of the packet 💀
Quick heads up 👀 the spec uses "fingerprint" and "signature" interchangeably. That's because what the fingerprint part looks like depends on which algorithm is used to insure the payload integrity. Also, whatever security you pick never implies encryption (the performance hit is real, bestie 😭). It's only fingerprint or signature generation and validation.
An implementation MUST implement at least both sending and receiving HMAC-SHA256 fingerprints as defined in Section 10.2 so everyone can interoperate 🫶 but it MAY use 'undefined_securitykey_id' by default.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | RIFT destination port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UDP Length | UDP Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Outer Security Envelope Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RIFT MAGIC | Packet Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | RIFT Major | Outer Key ID | Fingerprint |
| | Version | | Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Security Fingerprint covers all following content ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Weak Nonce Local | Weak Nonce Remote |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Remaining TIE Lifetime (all 1s in case of LIE) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
TIE Origin Security Envelope Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TIE Origin Key ID | Fingerprint |
| | Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Security Fingerprint covers all following content ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Serialized RIFT Model Object
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Serialized RIFT Model Object ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 34: Security Envelope
RIFT MAGIC: 16 bits 🪄 This is a constant value of 0xA1F7 that makes RIFT packets easy to classify, no matter which UDP port they use ✨
Packet Number: 16 bits 🔢 This is an optional, per-adjacency, per-packet type number, set using the sequence number arithmetic defined in Appendix A. If the arithmetic in Appendix A is not used, the node MUST set the value to undefined_packet_number. The number can help detect losses and misordering in flooding, either for operational purposes or so an implementation can adjust flooding behavior to current link or buffer quality 📊 It MUST NOT be used to discard packets or to validate their correctness 🙅♀️ Packet numbers are incremented on each interface, and within that, for each type of packet independently. This lets an implementation parallelize packet generation and processing for different types, if that's the vibe 🧚
RIFT Major Version: 8 bits 🏷️ This value MUST be set to "protocol_major_version", which is defined in the schema and used to serialize the object contained. It lets you check whether protocol versions are compatible on both sides, i.e., which schema version is needed to decode the serialized object. An implementation MUST drop packets with unexpected values and MAY report a problem 🚨 How an implementation may negotiate the schema's major version is outside the scope of this document, so don't look for it here 🙃
Outer Key ID: 8 bits 🗝️ This is a simple, unstructured value that acts as indirection into a structure holding an algorithm and any related secrets needed to validate any provided outer security fingerprint or signature. The value undefined_securitykey_id means that no valid fingerprint was computed or provided. Otherwise, one of the algorithms in Section 10.2 MUST be used to compute the fingerprint. This key ID scope is local to the nodes on both ends of the adjacency 🤝
TIE Origin Key ID: 24 bits 🗝️ This is a simple, unstructured value that acts as indirection into a structure holding an algorithm and any related secrets needed to validate any provided inner security fingerprint or signature. The value undefined_securitykey_id means that no valid fingerprint was computed. Otherwise, one of the algorithms in Section 10.2 MUST be used to compute the fingerprint. This key ID scope is global to the RIFT instance, since it may imply the originator of the TIE, so the contained object doesn't have to be deserialized to find out who the originator is 👑
Fingerprint Length: 8 bits 📏 This is the length in 32-bit multiples of the following fingerprint, not including lifetime or weak nonces. It lets the structure be navigated when an unknown key type is present. To clarify, a common corner case is when this value is set to 0, which signifies an empty (0 bytes long) security fingerprint 🫥
Security Fingerprint: 32 bits * Fingerprint Length 🔐 This is a signature computed over all the data that follows it. If the significant bits of the fingerprint are fewer than the 32-bit padded length, then the significant bits MUST be left aligned, and the remaining bits on the right are padded with 0s. When using Public Key Infrastructure (PKI), the originating node uses its private key to create the signature. The original packet can then be verified, provided the public key is shared and current. How keys are negotiated, distributed, or rolled over is outside the scope of this document 🗝️
Remaining TIE Lifetime: 32 bits ⏳ For anything but TIEs, this field MUST be set to all ones, and the Origin Security Envelope Header MUST NOT be present in the packet. For TIEs, this field represents the remaining lifetime of the TIE, and the Origin Security Envelope Header MUST be present in the packet 💖
Weak Nonce Local: 16 bits 🎲 This is the local Weak Nonce of the adjacency, as advertised in LIEs.
Weak Nonce Remote: 16 bits 🎲 This is the remote Weak Nonce of the adjacency, as received in LIEs.
TIE Origin Security Envelope Header: It MUST be present if and only if the Remaining TIE Lifetime field is not all ones 🎀 It carries the originator's key ID and the corresponding fingerprint of the object, which protects the TIE from modification during flooding. This ensures origin validation and integrity, but it does not provide validation of a chain of trust 🦋
Observe that, due to the schema migration rules in Section 7, the contained model can always be decoded if the major version matches and the envelope integrity has been validated. That means the description of the TIE is available to flood it properly, including unknown TIE types 🌈
6.9.4. Weak Nonces ✨
okay so this part is all about nonces 🫶 the protocol uses two 16-bit nonces to salt the signatures it generates. technically "nonce" is a bit of a loose term here, bestie, because RIFT nonces aren't changed on every single packet like they usually are in cryptography. 💅 instead they get changed often enough that replay attacks basically can't keep up, so from this point on they're called "weak" nonces 🥺
any implementation that uses an outer key ID other than undefined_securitykey_id MUST generate and wrap around its local nonces properly. it SHOULD do this even if it's not using any algorithm from Section 10.2 🌸 when a nonce increment lands on the _undefined_nonce_ value, it MUST increment again right away 🙈 every implementation MUST reflect the neighbor's nonces, no exceptions bestie 👑
an implementation SHOULD increment a chosen nonce on every LIE FSM transition that ends up in a different state from the previous one. it MUST increment its nonce at least every _nonce_regeneration_interval_ if it's using any algorithm in Section 10.2 🦋 this is what lets implementations stay efficient without opening a significant security risk, iykyk 🧚
when flooding TIEs, the implementation MUST use recent nonces (meaning within the allowed difference) that were reflected in the LIE exchange 💖 the schema sets _maximum_valid_nonce_delta_, which is the biggest allowed nonce difference between a packet and the reflected nonces in the LIEs. any packet with nonces that deviate more than the allowed delta MUST be discarded, and it has to happen without any signature computation so attackers can't spam you into a computation load attack 😭🔥
the delta can be a negative or positive difference that a mirrored nonce can drift from the local value and still count as valid 🌈 now here's the tea: if nonces only change at the maximum interval on both sides instead of every packet, that statistically opens up a _maximum_valid_nonce_delta_/2 window for replays of identical LIEs, TIEs, and TI(x)Es 💀 the interval also can't be too small, because the LIE FSM can change states pretty fast during ZTP without sending any LIEs, and UDP can both lose and misorder packets 🍓
In cases where a secure implementation doesn't receive signatures, or it gets undefined nonces from a neighbor (which tells you that neighbor doesn't support or verify signatures), it's up to local policy how those packets get treated 🎀 a secure implementation MAY refuse to form an adjacency with an implementation that isn't advertising signatures or valid nonces. it MAY also keep signing its own local packets while accepting the neighbor's packets without any further security validation 🫣
there's one necessary exception, though ✨ an implementation MUST advertise the remote nonce value as _undefined_nonce_ when the FSM is not in _TwoWay_ or _ThreeWay_ state, and it MUST accept an _undefined_nonce_ for its local nonce value on packets in any state other than _ThreeWay_ 🧚
as an optional optimization, an implementation MAY send one LIE with a previously negotiated neighbor's nonce to try to speed up the neighbor's transition from _ThreeWay_ to _OneWay_. after that one, it MUST revert to sending _undefined_nonce_ 🍓💜
6.9.5. Lifetime 🌷
reflooding the same TIE version quickly with tiny variations in its lifetime can lead to way too many security fingerprint computations 😩 to avoid that, the application generating fingerprints for flooded TIEs MAY round the lifetime value down to the next _rounddown_lifetime_interval_ in the packet header, so it can reuse previous computation results ✨ TIEs flooded with these rounded lifetimes only limit the computations needed during transitions that lead to advertising the same TIEs with the same information within a short period of time, bestie 🦋
6.9.6. Security Association Changes 🎀
no mechanism is specified for converting a security envelope for the same key ID from one algorithm to another once the envelope is operational 🙅♀️ the recommended procedure for switching to a new algorithm is to take the adjacency down, make the necessary changes to the secret and algorithm used by that key ID, and then bring the adjacency back up 🫶 obviously an implementation MAY choose to stop verifying the security envelope for the duration of the algorithm change so the adjacency can stay up. but this creates a security vulnerability window, so such a rollover SHOULD NOT be recommended 💀 other approaches, like accepting multiple algorithms for the same key ID for a configured time window, are possible, but they're in the realm of implementation choices rather than protocol specification 🌈
7. Information Elements Schema 💅
this section introduces the schema for information elements 💖 the Interface Description Language (IDL) is Thrift [thrift].
on schema changes that:
- change field numbers or
- add new required fields or
- remove any fields or
- change lists into sets, unions into structures or
- change multiplicity of fields or
- change type or name of any field or
- change data types of the type of any field or
- add, change or remove a default value of any existing field or
- remove or change any defined constant or constant value or
- change any enumeration type except extending
common.TIETypeType(use of enumeration types is generally discouraged) or - add new TIE type to _TIETypeType_ with flooding scope different from prefix TIE flooding scope
the major version of the schema MUST increase. all other changes MUST increase the minor version within the same major 🔥
introducing an optional field does not cause a major version increase, even if the fields inside the structure are optional with defaults ✨
all signed integers, as forced by Thrift [thrift] support, must be cast for internal purposes to equivalent unsigned values without discarding the signedness bit 🧚 an implementation SHOULD try to avoid using the signedness bit when generating values 🫶
the schema is normative 👑
7.1. Backwards-Compatible Extension of Schema 🦋
the set of rules in Section 7 guarantees that every decoder can process serialized content generated by a higher minor version of the schema, so the protocol can progress without a 'flag-day' 🎉 contrary to that, content serialized using a major version X is not expected to be decodable by any implementation using a decoder for a model with a major version lower than X 💀 schema negotiation and translation within RIFT is outside the scope of this document 🙏
additionally, based on the propagated minor version in encoded content and added optional node capabilities, new TIE types or even de facto mandatory fields can be introduced without progressing the major version 🌸 the catch is that only nodes supporting those new extensions would decode them 👀 given the model is encoded at the source and never re-encoded, flooding through nodes that don't understand any new extensions will preserve the corresponding fields 💖 but it's important to understand that a higher minor version of a schema does not guarantee that capabilities introduced in lower minors of the same major are supported 🙈 the _node_capabilities_ field is used to indicate which capabilities are supported 🍓
specifically, the schema SHOULD add elements to the _NodeCapabilities_ field's future capabilities to indicate whether it will support interpretation of schema extensions on the same major revision if they are present 🫶 such fields MUST be optional and have an implicit or explicit false default value 🔥 if a future capability changes route selection or generates conditions that cause packet loss when some nodes don't support it, then a major version increment will be unavoidable 😩 _NodeCapabilities_ shown in a LIE MUST match the capabilities shown in the Node TIEs, otherwise the behavior is unspecified 🤷♀️ a node that detects the mismatch SHOULD generate a notification 🔔
alternately or additionally, new optional fields can be introduced into, e.g., _NodeTIEElement_, if a special field is chosen to indicate via its presence that an optional feature is enabled 🌈 (capability to support a feature doesn't necessarily mean the feature is actually configured and operational, bestie 💅)
to support new TIE types without increasing the major version, _TIEElement_ can be extended with new optional elements for new 'common.TIETypeType' values, as long as the scope of the new TIE matches the prefix TIE scope ✨ if it's necessary to understand whether all nodes can parse the new TIE type, a node capability MUST be added in _NodeCapabilities_ to prevent a non-homogenous network 🧚💜
7.2. common.thrift 💖
okay so this file is basically the shared dictionary for RIFT, and it's giving "everyone agrees on what these words mean" energy 🫶 It references [RFC5837], [RFC5880], and [RFC6550] 🌸
/**
Thrift file with common definitions for RIFT
*/
namespace py common
/** @note MUST be interpreted in implementation as unsigned 64 bits.
*/
typedef i64 SystemIDType
typedef i32 IPv4Address
typedef i32 MTUSizeType
/** @note MUST be interpreted in implementation as unsigned
rolling over number */
typedef i64 SeqNrType
/** @note MUST be interpreted in implementation as unsigned */
typedef i32 LifeTimeInSecType
/** @note MUST be interpreted in implementation as unsigned */
typedef i8 LevelType
typedef i16 PacketNumberType
/** @note MUST be interpreted in implementation as unsigned */
typedef i32 PodType
/** @note MUST be interpreted in implementation as unsigned.
/** this has to be long enough to accommodate prefix */
typedef binary IPv6Address
/** @note MUST be interpreted in implementation as unsigned */
typedef i16 UDPPortType
/** @note MUST be interpreted in implementation as unsigned */
typedef i32 TIENrType
/** @note MUST be interpreted in implementation as unsigned
This is carried in the security envelope and must
hence fit into 8 bits. */
typedef i8 VersionType
/** @note MUST be interpreted in implementation as unsigned */
typedef i16 MinorVersionType
/** @note MUST be interpreted in implementation as unsigned */
typedef i32 MetricType
/** @note MUST be interpreted in implementation as unsigned
and unstructured */
typedef i64 RouteTagType
/** @note MUST be interpreted in implementation as unstructured
label value */
typedef i32 LabelType
/** @note MUST be interpreted in implementation as unsigned */
typedef i32 BandwidthInMegaBitsType
/** @note Key Value key ID type */
typedef i32 KeyIDType
/** node local, unique identification for a link (interface/tunnel/
* etc., basically anything RIFT runs on). This is kept
* at 32 bits so it aligns with BFD (RFC 5880) discriminator size.
*/
typedef i32 LinkIDType
/** @note MUST be interpreted in implementation as unsigned,
especially since we have the /128 IPv6 case. */
typedef i8 PrefixLenType
/** timestamp in seconds since the epoch */
typedef i64 TimestampInSecsType
/** security nonce.
@note MUST be interpreted in implementation as rolling
over unsigned value */
typedef i16 NonceType
/** LIE FSM holdtime type */
typedef i16 TimeIntervalInSecType
/** Transaction ID type for prefix mobility as specified by RFC 6550,
value MUST be interpreted in implementation as unsigned */
typedef i8 PrefixTransactionIDType
/** Timestamp per IEEE 802.1AS, all values MUST be interpreted in
implementation as unsigned. */
struct IEEE802_1ASTimeStampType {
1: required i64 AS_sec;
2: optional i32 AS_nsec;
}
/** generic counter type */
typedef i64 CounterType
/** Platform Interface Index type, i.e., index of interface on
hardware, can be used, e.g., with RFC 5837 */
typedef i32 PlatformInterfaceIndex
/** Flags indicating node configuration in case of ZTP.
*/
enum HierarchyIndications {
/** forces level to 'leaf_level' and enables
according procedures */
leaf_only = 0,
/** forces level to 'leaf_level' and enables
according procedures */
leaf_only_and_leaf_2_leaf_procedures = 1,
/** forces level to 'top_of_fabric' and enables according
procedures */
top_of_fabric = 2,
}
const PacketNumberType undefined_packet_number = 0
/** used when node is configured as top of fabric in ZTP.*/
const LevelType top_of_fabric_level = 24
/** default bandwidth on a link */
const BandwidthInMegaBitsType default_bandwidth = 100
/** fixed leaf level when ZTP is not used */
const LevelType leaf_level = 0
const LevelType default_level = leaf_level
const PodType default_pod = 0
const LinkIDType undefined_linkid = 0
/** invalid key for key value */
const KeyIDType invalid_key_value_key = 0
/** default distance used */
const MetricType default_distance = 1
/** any distance larger than this will be considered infinity */
const MetricType infinite_distance = 0x7FFFFFFF
/** represents invalid distance */
const MetricType invalid_distance = 0
const bool overload_default = false
const bool flood_reduction_default = true
/** default LIE FSM LIE TX interval time */
const TimeIntervalInSecType default_lie_tx_interval = 1
/** default LIE FSM holddown time */
const TimeIntervalInSecType default_lie_holdtime = 3
/** multiplier for default_lie_holdtime to
holddown multiple neighbors */
const i8 multiple_neighbors_lie_holdtime_multiplier = 4
/** default ZTP FSM holddown time */
const TimeIntervalInSecType default_ztp_holdtime = 1
/** by default LIE levels are ZTP offers */
const bool default_not_a_ztp_offer = false
/** by default everyone is repeating flooding */
const bool default_you_are_flood_repeater = true
/** 0 is illegal for System IDs */
const SystemIDType IllegalSystemID = 0
/** empty set of nodes */
const set<SystemIDType> empty_set_of_nodeids = {}
/** default lifetime of TIE is one week */
const LifeTimeInSecType default_lifetime = 604800
/** default lifetime when TIEs are purged is 5 minutes */
const LifeTimeInSecType purge_lifetime = 300
/** optional round down interval when
* TIEs are sent with security signatures
* to prevent excessive computation.
*/
const LifeTimeInSecType rounddown_lifetime_interval = 60
/** any 'TieHeader' that has a smaller lifetime difference
than this constant is equal (if other fields equal). */
const LifeTimeInSecType lifetime_diff2ignore = 400
/** default UDP port to run LIEs on */
const UDPPortType default_lie_udp_port = 914
/** default UDP port to receive TIEs on,
which can be peer specific */
const UDPPortType default_tie_udp_flood_port = 915
/** default MTU link size to use */
const MTUSizeType default_mtu_size = 1400
/** default link being BFD capable */
const bool bfd_default = true
/** type used to target nodes with key value */
typedef i64 KeyValueTargetType
/** default target for key value are all nodes. */
const KeyValueTargetType keyvaluetarget_default = 0
/** value for _all leaves_ addressing.
Represented by all bits set. */
const KeyValueTargetType keyvaluetarget_all_south_leaves = -1
/** undefined nonce, equivalent to missing nonce */
const NonceType undefined_nonce = 0;
/** outer security key ID, MUST be interpreted as in implementation
as unsigned */
typedef i8 OuterSecurityKeyID
/** security key ID, MUST be interpreted as in implementation
as unsigned */
typedef i32 TIESecurityKeyID
/** undefined key */
const TIESecurityKeyID undefined_securitykey_id = 0;
/** Maximum delta (negative or positive) that a mirrored nonce can
deviate from local value to be considered valid. */
const i16 maximum_valid_nonce_delta = 5;
const TimeIntervalInSecType nonce_regeneration_interval = 300;
/** Direction of TIEs. */
enum TieDirectionType {
Illegal = 0,
South = 1,
North = 2,
DirectionMaxValue = 3,
}
/** Address family type. */
enum AddressFamilyType {
Illegal = 0,
AddressFamilyMinValue = 1,
IPv4 = 2,
IPv6 = 3,
AddressFamilyMaxValue = 4,
}
/** IPv4 prefix type. */
struct IPv4PrefixType {
1: required IPv4Address address;
2: required PrefixLenType prefixlen;
}
/** IPv6 prefix type. */
struct IPv6PrefixType {
1: required IPv6Address address;
2: required PrefixLenType prefixlen;
}
/** IP address type. */
union IPAddressType {
/** Content is IPv4 */
1: optional IPv4Address ipv4address;
/** Content is IPv6 */
2: optional IPv6Address ipv6address;
}
/** Prefix advertisement.
@note: For interface
addresses, the protocol can propagate the address part beyond
the subnet mask and on reachability computation that has to
be normalized. The non-significant bits can be used
for operational purposes.
*/
union IPPrefixType {
1: optional IPv4PrefixType ipv4prefix;
2: optional IPv6PrefixType ipv6prefix;
}
/** Sequence of a prefix in case of move.
*/
struct PrefixSequenceType {
1: required IEEE802_1ASTimeStampType timestamp;
/** Transaction ID set by the client in, e.g., 6LoWPAN. */
2: optional PrefixTransactionIDType transactionid;
}
/** Type of TIE.
*/
enum TIETypeType {
Illegal = 0,
TIETypeMinValue = 1,
/** first legal value */
NodeTIEType = 2,
PrefixTIEType = 3,
PositiveDisaggregationPrefixTIEType = 4,
NegativeDisaggregationPrefixTIEType = 5,
PGPrefixTIEType = 6,
KeyValueTIEType = 7,
ExternalPrefixTIEType = 8,
PositiveExternalDisaggregationPrefixTIEType = 9,
TIETypeMaxValue = 10,
}
/** RIFT route types.
@note: The only purpose of those values is to introduce an
ordering, whereas an implementation can internally choose
any other values as long the ordering is preserved.
*/
enum RouteType {
Illegal = 0,
RouteTypeMinValue = 1,
/** First legal value. */
/** Discard routes are most preferred */
Discard = 2,
/** Local prefixes are directly attached prefixes on the
* system, such as interface routes.
*/
LocalPrefix = 3,
/** Advertised in S-TIEs */
SouthPGPPrefix = 4,
/** Advertised in N-TIEs */
NorthPGPPrefix = 5,
/** Advertised in N-TIEs */
NorthPrefix = 6,
/** Externally imported north */
NorthExternalPrefix = 7,
/** Advertised in S-TIEs, either normal prefix or positive
disaggregation */
SouthPrefix = 8,
/** Externally imported south */
SouthExternalPrefix = 9,
/** Negative, transitive prefixes are least preferred */
NegativeSouthPrefix = 10,
RouteTypeMaxValue = 11,
}
enum KVTypes {
Experimental = 1,
WellKnown = 2,
OUI = 3,
}7.3. encoding.thrift 🎀
this one is the packet encoding file, so it's where the protocol version lives. it includes the common file and sets the schema version numbers 🧚
/**
Thrift file for packet encodings for RIFT
*/
include "common.thrift"
namespace py encoding
/** Represents protocol encoding schema major version */
const common.VersionType protocol_major_version = 8
/** Represents protocol encoding schema minor version */
const common.MinorVersionType protocol_minor_version = 0okay bestie, the schema keeps going and it is giving structs 💅🎀
/** Common RIFT packet header. */
struct PacketHeader {
/** Major version of protocol. */
1: required common.VersionType major_version =
protocol_major_version;
/** Minor version of protocol. */
2: required common.MinorVersionType minor_version =
protocol_minor_version;
/** Node sending the packet, in case of LIE/TIRE/TIDE
also the originator of it. */
3: required common.SystemIDType sender;
/** Level of the node sending the packet, required on everything
except LIEs. Lack of presence on LIEs indicates
UNDEFINED_LEVEL and is used in ZTP procedures.
*/
4: optional common.LevelType level;
}
/** Prefix community. */
struct Community {
/** Higher order bits */
1: required i32 top;
/** Lower order bits */
2: required i32 bottom;
}
/** Neighbor structure. */
struct Neighbor {
/** System ID of the originator. */
1: required common.SystemIDType originator;
/** ID of remote side of the link. */
2: required common.LinkIDType remote_id;
}
/** Capabilities the node supports. */
struct NodeCapabilities {
/** Must advertise supported minor version dialect that way. */
1: required common.MinorVersionType protocol_minor_version =
protocol_minor_version;
/** indicates that node supports flood reduction. */
2: optional bool flood_reduction =
common.flood_reduction_default;
/** indicates place in hierarchy, i.e., top of fabric or
leaf only (in ZTP) or support for L2L
procedures. */
3: optional common.HierarchyIndications hierarchy_indications;
}
/** Link capabilities. */
struct LinkCapabilities {
/** Indicates that the link is supporting BFD. */
1: optional bool bfd =
common.bfd_default;
/** Indicates whether the interface will support IPv4
forwarding. */
2: optional bool ipv4_forwarding_capable =
true;
}
/** RIFT LIE Packet.
@note: This node's level is already included on the packet header.
*/
struct LIEPacket {
/** Node or adjacency name. */
1: optional string name;
/** Local link ID. */
2: required common.LinkIDType local_id;
/** UDP port to which we can receive flooded TIEs. */
3: required common.UDPPortType flood_port =
common.default_tie_udp_flood_port;
/** Layer 2 MTU, used to discover mismatch. */
4: optional common.MTUSizeType link_mtu_size =
common.default_mtu_size;
/** Local link bandwidth on the interface. */
5: optional common.BandwidthInMegaBitsType
link_bandwidth = common.default_bandwidth;
/** Reflects the neighbor once received to provide
3-way connectivity. */
6: optional Neighbor neighbor;
/** Node's PoD. */
7: optional common.PodType pod =
common.default_pod;
/** Node capabilities supported. */
10: required NodeCapabilities node_capabilities;
/** Capabilities of this link. */
11: optional LinkCapabilities link_capabilities;
/** Required holdtime of the adjacency, i.e., for how long a
period adjacency should be kept up without valid LIE
reception. */
12: required common.TimeIntervalInSecType
holdtime = common.default_lie_holdtime;
/** Optional, unsolicited, downstream assigned locally significant
label value for the adjacency. */
13: optional common.LabelType label;
/** Indicates that the level on the LIE must not be used
to derive a ZTP level by the receiving node. */
21: optional bool not_a_ztp_offer =
common.default_not_a_ztp_offer;
/** Indicates to northbound neighbor that it should
be reflooding TIEs received from this node to achieve flood
reduction and balancing for northbound flooding. */
22: optional bool you_are_flood_repeater =
common.default_you_are_flood_repeater;
/** Indicates to neighbor to flood node TIEs only and slow down
all other TIEs. Ignored when received from southbound
neighbor. */
23: optional bool you_are_sending_too_quickly =
false;
/** Instance name in case multiple RIFT instances running on same
interface. */
24: optional string instance_name;
/** It provides the optional ID of the fabric configured. This
MUST match the information advertised on the node element. */
35: optional common.FabricIDType fabric_id =
common.default_fabric_id;
}
/** LinkID pair describes one of parallel links between two nodes. */
struct LinkIDPair {
/** Node-wide unique value for the local link. */
1: required common.LinkIDType local_id;
/** Received remote link ID for this link. */
2: required common.LinkIDType remote_id;
/** Describes the local interface index of the link. */
10: optional common.PlatformInterfaceIndex
platform_interface_index;
/** Describes the local interface name. */
11: optional string platform_interface_name;
/** Indicates whether the link is secured, i.e., protected by
outer key, absence of this element means no indication,
undefined outer key means not secured. */
12: optional common.OuterSecurityKeyID
trusted_outer_security_key;
/** Indicates whether the link is protected by established
BFD session. */
13: optional bool bfd_up;
/** Optional indication which address families are up on the
interface */
14: optional set<common.AddressFamilyType>
address_families;
}
/** Unique ID of a TIE. */
struct TIEID {
/** direction of TIE */
1: required common.TieDirectionType direction;
/** indicates originator of the TIE */
2: required common.SystemIDType originator;
/** type of the tie */
3: required common.TIETypeType tietype;
/** number of the tie */
4: required common.TIENrType tie_nr;
}
/** Header of a TIE. */
struct TIEHeader {
/** ID of the tie. */
2: required TIEID tieid;
/** Sequence number of the tie. */
3: required common.SeqNrType seq_nr;
/** Absolute timestamp when the TIE was generated. */
10: optional common.IEEE802_1ASTimeStampType origination_time;
/** Original lifetime when the TIE was generated. */
12: optional common.LifeTimeInSecType origination_lifetime;
}
/** Header of a TIE as described in TIRE/TIDE.
*/
struct TIEHeaderWithLifeTime {
1: required TIEHeader header;
/** Remaining lifetime. */
2: required common.LifeTimeInSecType remaining_lifetime;
}
/** TIDE with *sorted* TIE headers. */
struct TIDEPacket {
/** First TIE header in the TIDE packet. */
1: required TIEID start_range;
/** Last TIE header in the TIDE packet. */
2: required TIEID end_range;
/** _Sorted_ list of headers. */
3: required list<TIEHeaderWithLifeTime>
headers;
}
/** TIRE packet */
struct TIREPacket {
1: required set<TIEHeaderWithLifeTime>
headers;
}
/** neighbor of a node */
struct NodeNeighborsTIEElement {
/** level of neighbor */
1: required common.LevelType level;
/** Cost to neighbor. Ignore anything equal/larger than
'infinite_distance' or equal 'invalid_distance' */
3: optional common.MetricType cost
= common.default_distance;
/** can carry description of multiple parallel links in a TIE */
4: optional set<LinkIDPair>
link_ids;
/** total bandwidth to neighbor as sum of all parallel links */
5: optional common.BandwidthInMegaBitsType
bandwidth = common.default_bandwidth;
}
/** Indication flags of the node. */
struct NodeFlags {
/** Indicates that node is in overload, do not transit traffic
through it. */
1: optional bool overload = common.overload_default;
}
/** Description of a node. */
struct NodeTIEElement {
/** Level of the node. */
1: required common.LevelType level;
/** Node's neighbors. Multiple node TIEs can carry disjoint sets
of neighbors. */
2: required map<common.SystemIDType,
NodeNeighborsTIEElement> neighbors;
/** Capabilities of the node. */
3: required NodeCapabilities capabilities;
/** Flags of the node. */
4: optional NodeFlags flags;
/** Optional node name for easier operations. */
5: optional string name;
/** PoD to which the node belongs. */
6: optional common.PodType pod;
/** Optional startup time of the node */
7: optional common.TimestampInSecsType startup_time;
/** If any local links are miscabled, this indication is
flooded. */
10: optional set<common.LinkIDType>
miscabled_links;
/** ToFs in the same plane. Only carried by ToF. Multiple Node
TIEs can carry disjoint sets of ToFs that MUST be joined to
form a single set. */
12: optional set<common.SystemIDType>
same_plane_tofs;
/** It provides the optional ID of the fabric configured */
20: optional common.FabricIDType fabric_id =
common.default_fabric_id;
}
/** Attributes of a prefix. */
struct PrefixAttributes {
/** Distance of the prefix. */
2: required common.MetricType metric
= common.default_distance;
/** Generic unordered set of route tags, can be redistributed
to other protocols or used within the context of real time
analytics. */
3: optional set<common.RouteTagType>
tags;
/** Monotonic clock for mobile addresses. */
4: optional common.PrefixSequenceType monotonic_clock;
/** Indicates if the prefix is a node loopback. */
6: optional bool loopback = false;
/** Indicates that the prefix is directly attached. */
7: optional bool directly_attached = true;
/** Link to which the address belongs to. */
10: optional common.LinkIDType from_link;
/** Optional, per-prefix significant label. */
12: optional common.LabelType label;
}
/** TIE carrying prefixes */
struct PrefixTIEElement {
/** Prefixes with the associated attributes. */
1: required map<common.IPPrefixType, PrefixAttributes> prefixes;
}
/** Defines the targeted nodes and the value carried. */
struct KeyValueTIEElementContent {
1: optional common.KeyValueTargetType targets =
common.keyvaluetarget_default;
2: optional binary value;
} /** Generic key value pairs. */
struct KeyValueTIEElement {
1: required map<common.KeyIDType, KeyValueTIEElementContent>
keyvalues;
}
/** Single element in a TIE. */
union TIEElement {
/** Used in case of enum common.TIETypeType.NodeTIEType. */
1: optional NodeTIEElement node;
/** Used in case of enum common.TIETypeType.PrefixTIEType. */
2: optional PrefixTIEElement prefixes;
/** Positive prefixes (always southbound). */
3: optional PrefixTIEElement positive_disaggregation_prefixes;
/** Transitive, negative prefixes (always southbound) */
5: optional PrefixTIEElement negative_disaggregation_prefixes;
/** Externally reimported prefixes. */
6: optional PrefixTIEElement external_prefixes;
/** Positive external disaggregated prefixes (always
southbound). */
7: optional PrefixTIEElement
positive_external_disaggregation_prefixes;
/** Key-Value store elements. */
9: optional KeyValueTIEElement keyvalues;
}
/** TIE packet */
struct TIEPacket {
1: required TIEHeader header;
2: required TIEElement element;
}
/** Content of a RIFT packet. */
union PacketContent {
1: optional LIEPacket lie;
2: optional TIDEPacket tide;
3: optional TIREPacket tire;
4: optional TIEPacket tie;
}
/** RIFT packet structure. */
struct ProtocolPacket {
1: required PacketHeader header;
2: required PacketContent content;
}8. Further Details on Implementation ✨
8.1. Considerations for Leaf-Only Implementation
okay so RIFT can be stretched, and is actually meant to be, all the way down to the lowest level of the IP fabric 🧚 so it can scoop up ToRs or even servers. Those guys only ever run as leaves, so it's worth noting that a leaf-only version is way simpler to implement and needs much less resources 💅:
- Leaf nodes only need to keep a multipath default route when things are normal 🫶. But if there's a catastrophic partition, leaf nodes SHOULD be able to fit all the leaf routes in their own PoD so no traffic gets lost 😭.
- Leaf nodes only hold their own North TIEs and the South TIEs of the level 1 nodes they're connected to 🌸.
- Leaf nodes don't have to support any kind of disaggregation computation or propagation. no stress 🦋.
- Leaf nodes are not required to support the overload flag 💜.
- Leaf nodes don't need to originate S-TIEs unless you want the optional L2L features 🍓.
8.2. Considerations for Spine Implementation
ok this one's giving "the non-ToF nodes have it easy but also have to listen up" 👑. Nodes that aren't acting as ToF don't have to hunt down fallen leaves by comparing reachable destinations with their peers, so they don't need to run the disaggregated route computation that comes from that discovery 🙈. Still, non-ToF nodes need to respect the disaggregated routes advertised from the north 🔥. And when a fallen leaf loses all its parents, spine nodes need to generate southbound disaggregated routes for it, because that's literally their job now 💀.
9. Security Considerations 🛡️
9.1. General
bestie, we have to talk about the attack vibes 😭. Picture a router that reboots over and over, changing its System ID each time, and floods the network with stale TIEs, or TIEs sent with super long lifetimes that never get cleaned up after the routes vanish. Those attack vectors aren't unique to RIFT 🙄. Given how huge memory footprints are today, those attacks should be pretty benign. Otherwise, a node SHOULD have a strategy for discarding the contents of all TIEs that weren't in the SPF tree for a certain, configurable period of time 🧹. The protocol is self-stabilizing and will advertise the presence of those TIEs to its neighbors, so they can be re-requested if a computation finds that it has an adjacency formed toward the System ID of the discarded TIEs 🔁.
The inner protection you set up with any mechanism in Section 10.2 guarantees the integrity of TIE content 🔐. When you combine it with the outer part of the envelope, using any of the mechanisms in Section 10.2, you also get protection against replay attacks ✨. If you only apply outer protection (meaning an outer key ID that's different from 'undefined_securitykey_id') to an adjacency using any mechanism in Section 10.2, then the integrity of the packet and replay protection only hold over the adjacency involved, in whichever configured directions 🫠. More considerations are in Sections 9.7 and 9.8 🦋.
9.2. Time to Live and Hop Limit Values
RIFT explicitly requires the use of a TTL/HL value of 1 or 255 when sending and receiving LIEs and TIEs, so implementors get to choose between the two 🎀.
Using a TTL/HL value of 255 does come with security concerns, but those risks are addressed in [RFC5082] 💅. Still, this approach may have difficulties with some forwarding implementations, like incorrectly processing TTL/HL or loops within the forwarding plane itself, etc. 😵💫
That's exactly why RIFT also lets implementations use a TTL/HL of 1 🧚. Attacks that exploit this by spoofing it from several hops away are possible, but they're exceptionally hard to pull off 🕵️. Replay attacks are another potential vector, but as the later security sections explain, RIFT is well protected against them if any mechanism in Section 10.2 is applied 🔒. Also, for link-local scoped multicast addresses used for LIE, a value of 1 is the more consistent choice 💖.
9.3. Malformed Packets
The protocol protects packets heavily through optional signatures and nonces 🛡️. So if malicious injected malformed or replayed packets are a possibility in your deployment, the algorithms in Section 10.2 must be applied 🚨.
Even with the security envelope, RIFT relies on Thrift encoders and decoders that are generated automatically from IDL 🤖. It's conceivable that errors in those encoders and decoders could be found and lead to delivery of corrupted packets, or to reception of packets that can't be decoded 😬. Misformatted packets normally make the decoder return an error to the caller, so the packet is basically unparsable and the only option is to discard it 🗑️. If the decoder ever gets forced to abort the protocol (the unlikely scenario), that's neither better nor worse than what other protocols do today 🤷♀️.
9.4. RIFT ZTP
Section 6.7 lays out a bunch of attack vectors in untrusted environments 🚩. The big one: nodes that oscillate their level offers, and nodes offering a ThreeWay adjacency with the highest possible level value and a very long holdtime, trying to put themselves "on top of the lattice" 👑. That would let them access the whole southbound topology, which is so not it 😤. Session authentication mechanisms are necessary in environments where this is possible, and RIFT provides the security envelope to handle it, if you want it, as long as any mechanism in Section 10.2 is deployed 🔐.
9.5. Lifetime
RIFT gets rid of lifetime modification and replay attack vectors by protecting the lifetime behind a signature computed over it, plus an additional nonce combination 💪. The result is that an attacker can't artificially shorten the remaining_lifetime 🙅♀️. This only applies if any mechanism in Section 10.2 is used ✨.
9.6. Packet Number
A packet number is an optional, defined value carried in the security envelope with no fingerprint protection, so it's vulnerable to replay and modification attacks 😩. Unlike nonces, this number must change on every packet, and signing it would be a super high cryptographic load 🥵. Honestly, the packet number attack vector is relatively benign 🌸. A man-in-the-middle changing the packet number will only affect operational validation tools and maybe some performance optimizations on flooding 🧃. An implementation that detects too many "fake losses" or "misorderings" caused by the attack on the packet number is expected to simply suppress further processing of it 🙅♀️.
9.7. Outer Fingerprint Attacks
Even when a mechanism in Section 10.2 is enabled to generate outer fingerprints, further attack considerations still apply 👀.
A node can try to inject LIE packets while observing a conversation on the wire, by using the observed outer key ID 🕵️♀️. It can't generate valid signatures if it changes the integrity of the message, though, so the only possible attack is DoS through excessive LIE validation, if any mechanism in Section 10.2 is used 😤.
A node can also try to replay previous LIEs with a changed state it recorded 🎞️. That's hard to replicate, since the nonce combination has to match the ongoing exchange, and it's limited to a single flap because both nodes advance their nonces if the adjacency state changes 🦋. Even in the most unlikely case, the attack length is limited because both sides periodically increase their nonces ⏱️.
Weak nonces aren't changed on every packet for performance reasons, so a conceivable attack by a man in the middle is to flood a receiving node with the maximum bandwidth of recently observed packets, both LIEs and TIEs 🌊. Where such attacks are likely, maximum_valid_nonce_delta and nonce_regeneration_interval can be implemented as configurable and set to small values 🔧. Heads up though: this will likely put a significant computational load on large fabrics under normal operation 🥲.
9.8. TIE Origin Fingerprint DoS Attacks
Even when a mechanism in Section 10.2 is enabled to generate inner fingerprints or signatures, further attack considerations apply 🧐.
If the inner fingerprint could be generated by a compromised node in the network other than the originator, based on shared secrets, the deployment must fall back on signatures that can be validated but not generated by any node except the originator 🔏. The key point is that only the originator can make them.
A compromised node in the network can try to brute force "fake TIEs" using other nodes' TIE origin key IDs without having the necessary secrets 🥷. The ultimate origin signature validation will fail in those cases and won't go further than the immediately peering nodes 🛑. Still, the resulting DoS attack seems unavoidable, since the TIE origin key ID is only protected by the (here assumed to be compromised) node 😭💀.
9.9. Host Implementations 🖥️✨
okay so it's honestly kinda giving that more and more RIFT devices are going to be RotH servers instead of dedicated networking boxes 💅 and bestie, servers come with a way wider software envelope and way more access, so they're also way more likely to get compromised and turn into an attack vector on the protocol 😭. hijacking prefixes to pull in traffic is a trust problem, and if the trust model gets breached (aka the server shows valid credentials to form an adjacency and issue TIEs), the protocol can't really fix that on its own 💀. it gets even more devious: a server could DoS (or even DDoS) the fabric by sending way too many LIE packets, flooding a ton of North TIEs, or pulling other resource overrun stunts 🙈. so a careful implementation that forms adjacencies to leaves should put threshold mechanisms in place and raise warnings when, for example, a leaf is advertising an excess number of TIEs or prefixes 🚨. on top of that, that kind of implementation could refuse any topology information except the node's own TIEs and the authenticated, reflected South Node TIEs at its own level 🛡️.
to isolate the possible attack vectors on the leaf as much as possible, a dedicated leaf-only implementation could run with zero configuration by:
- hard-coding a well-known adjacency key (which can always be rolled over by means of, e.g., a well-known key-value distributed from top of the fabric) 🔑
- hard-coding a leaf level value, and 🍃
- always setting the overload flag. 🚧
9.9.1. IPv4 Broadcast and IPv6 All-Routers Multicast Implementations 📡💜
Section 6.2 describes an optional implementation that supports LIE exchange over IPv4 broadcast addresses and/or the IPv6 all-routers multicast address. it's important to think about this one 🧐: if an implementation supports it, the attack surface widens, because LIEs may get propagated to devices outside of the intended RIFT topology. that can leave RIFT nodes more susceptible to the various attack vectors we already covered in this section 😬🫣.
10. IANA Considerations 📋🎀
so this part is all about the stuff IANA has to set up 💖. as detailed below, multicast addresses and standard port numbers have been assigned. there are also registries for the schema, and they've been created with initial values already filled in ✨.
10.1. Multicast and Port Numbers 🌸
in the "IPv4 Multicast Address Space" registry, the value 224.0.0.121 has been assigned for 'ALL_V4_RIFT_ROUTERS'. in the "IPv6 Multicast Address Space" registry, the value ff02::a1f7 has been assigned for 'ALL_V6_RIFT_ROUTERS' 🦋.
the following assignments have been made in the "Service Name and Transport Protocol Port Number Registry" 🧚:
RIFT LIE Port
Service Name: rift-lies
Port Number: 914
Transport Protocol: udp
Description: Routing in Fat Trees Link Information Element
Assignee: IESG (iesg@ietf.org)
Contact: IETF Chair (chair@ietf.org)
Reference: RFC 9692
RIFT TIE Port
Service Name: rift-ties
Port Number: 915
Transport Protocol: udp
Description: Routing in Fat Trees Topology Information Element
Assignee: IESG (iesg@ietf.org)
Contact: IETF Chair (chair@ietf.org)
Reference: RFC 969210.2. Registry for RIFT Security Algorithms 🔐💅
a brand new registry was created to hold the allowed RIFT security algorithms 👑. no particular enumeration values are necessary, because RIFT uses a key ID abstraction on packets. that means it doesn't disclose any info about the algorithm or the secrets used, and it only carries the resulting fingerprint or signature that protects the integrity of the data 🫶.
the registry applies the "Specification Required" policy per [RFC8126]. the designated expert should make sure the suggested algorithms represent the state of the art at a given point in time, and avoid introducing algorithms that don't bring enhanced security properties, or don't ensure such properties at a lower cost compared to existing registry entries 🧠.
+==========================+==========================+============+
| Name | Recommendation | Reference |
+==========================+==========================+============+
| HMAC-SHA256 | Simplest way to ensure | [SHA-2] |
| | integrity of | and |
| | transmissions across | [RFC2104] |
| | adjacencies when used as | |
| | outer keys and integrity | |
| | of TIEs when used as | |
| | inner keys. Recommended | |
| | for most interoperable | |
| | security protection. | |
+--------------------------+--------------------------+------------+
| HMAC-SHA512 | Same as HMAC-SHA256 with | [SHA-2] |
| | stronger protection. | and |
| | | [RFC2104] |
+--------------------------+--------------------------+------------+
| SHA256-RSASSA-PKCS1-v1_5 | Recommended for high | [RFC8017], |
| | security applications | Section |
| | where private keys are | 8.2 |
| | protected by according | |
| | nodes. Recommended as | |
| | well in case not only | |
| | integrity but origin | |
| | validation is necessary | |
| | for TIEs. Recommended | |
| | when adjacencies must be | |
| | protected without | |
| | disclosing the secrets | |
| | on both sides of the | |
| | adjacency. | |
+--------------------------+--------------------------+------------+
| SHA512-RSASSA-PKCS1-v1_5 | Same as SHA256-RSASSA- | [RFC8017] |
| | PKCS1-v1_5 with stronger | |
| | protection. | |
+--------------------------+--------------------------+------------+
Table 7: RIFT Security Algorithms10.3. Registries with Assigned Values for Schema Values 🗂️💖
this section requests registries that help govern the schema through the usual IANA registry procedures 🎀. the registry group "Routing in Fat Trees (RIFT)" holds the following registries. registry values are stored with their minimum and maximum version in which they're available. all values not provided are to be considered "Unassigned". the range of every registry is a 16-bit integer. allocation of new values is done via "Expert Review" only in the case of minor changes, per the rules in Section 7. all other allocations are done via "Specification Required" ✨.
in some cases, the registries don't contain necessary info like whether fields are optional or required, what units are used, or what datatype is involved. that info is encoded in the normative schema itself, using IDL syntax or the necessary type definitions and their names 🧩.
10.3.1. RIFTVersions Registry 🔢
this registry stores all RIFT protocol schema major and minor versions, including the reference to the document that introduced each version. it also means that if multiple documents extend the rift schema, they have to serialize using this registry to increase the minor or major versions sequentially 🌈.
+================+=====================+
| Schema Version | Reference |
+================+=====================+
| 8.0 | RFC 9692, Section 7 |
+----------------+---------------------+
Table 810.3.2. RIFTCommonAddressFamilyType Registry 🌍
this registry has the following initial values. in addition to the columns shown below, the IANA registry also includes Comment and Reference columns 🦋.
+=======+=======================+=====================+=============+
| Value | Name | Min. Schema | Max. Schema |
| | | Version | Version |
+=======+=======================+=====================+=============+
| 0 | Illegal | 8.0 | |
+-------+-----------------------+---------------------+-------------+
| 1 | AddressFamilyMinValue | 8.0 | |
+-------+-----------------------+---------------------+-------------+
| 2 | IPv4 | 8.0 | |
+-------+-----------------------+---------------------+-------------+
| 3 | IPv6 | 8.0 | |
+-------+-----------------------+---------------------+-------------+
| 4 | AddressFamilyMaxValue | 8.0 | |
+-------+-----------------------+---------------------+-------------+
Table 9: Address Family Type10.3.3. RIFTCommonHierarchyIndications Registry 🪜
this registry has the following initial values. in addition to the columns below, the IANA registry also includes Comment and Reference columns 💜.
+=======+======================================+=========+=========+
| Value | Name | Min. | Max. |
| | | Schema | Schema |
| | | Version | Version |
+=======+======================================+=========+=========+
| 0 | leaf_only | 8.0 | |
+-------+--------------------------------------+---------+---------+
| 1 | leaf_only_and_leaf_2_leaf_procedures | 8.0 | |
+-------+--------------------------------------+---------+---------+
| 2 | top_of_fabric | 8.0 | |
+-------+--------------------------------------+---------+---------+
Table 10: Flags Indicating Node Configuration in Case of ZTP10.3.4. RIFTCommonIEEE8021ASTimeStampType Registry ⏰
this registry has the following initial values. in addition to the columns below, the IANA registry also includes Comment and Reference columns 🌸.
the timestamp is per IEEE 802.1AS; all values MUST be interpreted in implementation as unsigned. 🔢✨
+=======+==========+=====================+=====================+
| Value | Name | Min. Schema Version | Max. Schema Version |
+=======+==========+=====================+=====================+
| 0 | Reserved | 8.0 | All Versions |
+-------+----------+---------------------+---------------------+
| 1 | AS_sec | 8.0 | |
+-------+----------+---------------------+---------------------+
| 2 | AS_nsec | 8.0 | |
+-------+----------+---------------------+---------------------+
Table 1110.3.5. RIFTCommonIPAddressType Registry ✨
okay so this registry is giving very clear vibes 💅 it has the following initial values bestie 🎀
+=======+=============+=====================+=============+=========+
| Value | Name | Min. Schema | Max. Schema | Comment |
| | | Version | Version | |
+=======+=============+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+-------------+---------------------+-------------+---------+
| 1 | ipv4address | 8.0 | | Content |
| | | | | is IPv4 |
+-------+-------------+---------------------+-------------+---------+
| 2 | ipv6address | 8.0 | | Content |
| | | | | is IPv6 |
+-------+-------------+---------------------+-------------+---------+
Table 12: IP Address Type10.3.6. RIFTCommonIPPrefixType Registry 🦋
this one has initial values too 🌸 and there's a little note that's SO important, so read it carefully 👀
| Note: For interface addresses the protocol can propagate the
| address part beyond the subnet mask and on reachability
| computation the non-significant bits have to be normalized.
| Those bits can be used for operational purposes.+=======+============+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+============+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+------------+---------------------+-------------+---------+
| 1 | ipv4prefix | 8.0 | | |
+-------+------------+---------------------+-------------+---------+
| 2 | ipv6prefix | 8.0 | | |
+-------+------------+---------------------+-------------+---------+
Table 13: Prefix Advertisement10.3.7. RIFTCommonIPv4PrefixType Registry 🌈
initial values, obviously 💖
+=======+===========+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+===========+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+-----------+---------------------+-------------+---------+
| 1 | address | 8.0 | | |
+-------+-----------+---------------------+-------------+---------+
| 2 | prefixlen | 8.0 | | |
+-------+-----------+---------------------+-------------+---------+
Table 14: IPv4 Prefix Type10.3.8. RIFTCommonIPv6PrefixType Registry ⭐
same vibe as the IPv4 one, just for v6 🧚
+=======+===========+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+===========+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+-----------+---------------------+-------------+---------+
| 1 | address | 8.0 | | |
+-------+-----------+---------------------+-------------+---------+
| 2 | prefixlen | 8.0 | | |
+-------+-----------+---------------------+-------------+---------+
Table 15: IPv6 Prefix Type10.3.9. RIFTCommonKVTypes Registry 🍓
this one has initial values bestie 💜
+=======+==============+=============+=============+=========+
| Value | Name | Min. Schema | Max. Schema | Comment |
| | | Version | Version | |
+=======+==============+=============+=============+=========+
| 0 | Unassigned | | | |
+-------+--------------+-------------+-------------+---------+
| 1 | Experimental | 8.0 | | |
+-------+--------------+-------------+-------------+---------+
| 2 | WellKnown | 8.0 | | |
+-------+--------------+-------------+-------------+---------+
| 3 | OUI | 8.0 | | |
+-------+--------------+-------------+-------------+---------+
Table 1610.3.10. RIFTCommonPrefixSequenceType Registry 🎀
initial values, and the transactionid row is a whole vibe 🌸
+=======+===============+=========+==========+===================+
| Value | Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=======+===============+=========+==========+===================+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+---------------+---------+----------+-------------------+
| 1 | timestamp | 8.0 | | |
+-------+---------------+---------+----------+-------------------+
| 2 | transactionid | 8.0 | | Transaction ID |
| | | | | set by client in, |
| | | | | e.g., 6LoWPAN. |
+-------+---------------+---------+----------+-------------------+
Table 17: Sequence of a Prefix in Case of Move10.3.11. RIFTCommonRouteType Registry 💅
this registry has initial values, and there's a note here that matters a lot 🫶
| Note: The only purpose of these values is to introduce an
| ordering, whereas an implementation can internally choose any
| other values as long the ordering is preserved.+=======+=====================+=============+=============+=========+
| Value | Name | Min. Schema | Max. | Comment |
| | | Version | Schema | |
| | | | Version | |
+=======+=====================+=============+=============+=========+
| 0 | Illegal | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 1 | RouteTypeMinValue | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 2 | Discard | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 3 | LocalPrefix | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 4 | SouthPGPPrefix | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 5 | NorthPGPPrefix | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 6 | NorthPrefix | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 7 | NorthExternalPrefix | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 8 | SouthPrefix | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 9 | SouthExternalPrefix | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 10 | NegativeSouthPrefix | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
| 11 | RouteTypeMaxValue | 8.0 | | |
+-------+---------------------+-------------+-------------+---------+
Table 18: RIFT Route Types10.3.12. RIFTCommonTIETypeType Registry 👑
this one has initial values, and it also has Comment and Reference columns in the real IANA registry, just not shown here 🙈
+=====+===========================================+=======+=======+
|Value|Name |Min. |Max. |
| | |Schema |Schema |
| | |Version|Version|
+=====+===========================================+=======+=======+
|0 |Illegal |8.0 | |
+-----+-------------------------------------------+-------+-------+
|1 |TIETypeMinValue |8.0 | |
+-----+-------------------------------------------+-------+-------+
|2 |NodeTIEType |8.0 | |
+-----+-------------------------------------------+-------+-------+
|3 |PrefixTIEType |8.0 | |
+-----+-------------------------------------------+-------+-------+
|4 |PositiveDisaggregationPrefixTIEType |8.0 | |
+-----+-------------------------------------------+-------+-------+
|5 |NegativeDisaggregationPrefixTIEType |8.0 | |
+-----+-------------------------------------------+-------+-------+
|6 |PGPrefixTIEType |8.0 | |
+-----+-------------------------------------------+-------+-------+
|7 |KeyValueTIEType |8.0 | |
+-----+-------------------------------------------+-------+-------+
|8 |ExternalPrefixTIEType |8.0 | |
+-----+-------------------------------------------+-------+-------+
|9 |PositiveExternalDisaggregationPrefixTIEType|8.0 | |
+-----+-------------------------------------------+-------+-------+
|10 |TIETypeMaxValue |8.0 | |
+-----+-------------------------------------------+-------+-------+
Table 19: Type of TIE10.3.13. RIFTCommonTieDirectionType Registry 🦋
this one has initial values, and the direction stuff is kinda important 🔥
+=======+===================+=============+=============+=========+
| Value | Name | Min. Schema | Max. Schema | Comment |
| | | Version | Version | |
+=======+===================+=============+=============+=========+
| 0 | Illegal | 8.0 | | |
+-------+-------------------+-------------+-------------+---------+
| 1 | South | 8.0 | | |
+-------+-------------------+-------------+-------------+---------+
| 2 | North | 8.0 | | |
+-------+-------------------+-------------+-------------+---------+
| 3 | DirectionMaxValue | 8.0 | | |
+-------+-------------------+-------------+-------------+---------+
Table 20: Direction of TIEs10.3.14. RIFTEncodingCommunity Registry ✨
Okay bestie, here are the starting values for this one 💅
+=======+==========+=====================+=============+============+
| Value | Name | Min. Schema | Max. Schema | Comment |
| | | Version | Version | |
+=======+==========+=====================+=============+============+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------+---------------------+-------------+------------+
| 1 | top | 8.0 | | Higher |
| | | | | order bits |
+-------+----------+---------------------+-------------+------------+
| 2 | bottom | 8.0 | | Lower |
| | | | | order bits |
+-------+----------+---------------------+-------------+------------+
Table 21: Prefix Community10.3.15. RIFTEncodingKeyValueTIEElement Registry 🎀
Here are the initial values for this registry, and they're giving clean and simple 🦋
+=======+===========+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+===========+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+-----------+---------------------+-------------+---------+
| 1 | keyvalues | 8.0 | | |
+-------+-----------+---------------------+-------------+---------+
Table 22: Generic Key Value Pairs10.3.16. RIFTEncodingKeyValueTIEElementContent Registry 🧚
Okay so, here are the starting values. This one defines the targeted nodes and the value being carried, so it's basically the who and the what 💖
+=======+==========+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+==========+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------+---------------------+-------------+---------+
| 1 | targets | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
| 2 | value | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
Table 2310.3.17. RIFTEncodingLIEPacket Registry 🦋
okay so this registry comes with a starter set of values already filled in 💅 there's a little note up top too: a node's level is already in the packet header, so it isn't repeated here 🫶
This registry has the following initial values.
| Note: This node's level is already included on the packet
| header.
+=====+=============================+=======+========+==============+
|Value| Name |Min. |Max. |Comment |
| | |Schema |Schema | |
| | |Version|Version | |
+=====+=============================+=======+========+==============+
|0 | Reserved |8.0 |All | |
| | | |Versions| |
+-----+-----------------------------+-------+--------+--------------+
|1 | name |8.0 | |Node or |
| | | | |adjacency |
| | | | |name. |
+-----+-----------------------------+-------+--------+--------------+
|2 | local_id |8.0 | |Local link |
| | | | |ID. |
+-----+-----------------------------+-------+--------+--------------+
|3 | flood_port |8.0 | |UDP port to |
| | | | |which we can |
| | | | |receive |
| | | | |flooded TIEs. |
+-----+-----------------------------+-------+--------+--------------+
|4 | link_mtu_size |8.0 | |Layer 2 MTU, |
| | | | |used to |
| | | | |discover |
| | | | |mismatch. |
+-----+-----------------------------+-------+--------+--------------+
|5 | link_bandwidth |8.0 | |Local link |
| | | | |bandwidth on |
| | | | |the |
| | | | |interface. |
+-----+-----------------------------+-------+--------+--------------+
|6 | neighbor |8.0 | |Reflects the |
| | | | |neighbor once |
| | | | |received to |
| | | | |provide 3-way |
| | | | |connectivity. |
+-----+-----------------------------+-------+--------+--------------+
|7 | pod |8.0 | |Node's PoD. |
+-----+-----------------------------+-------+--------+--------------+
|10 | node_capabilities |8.0 | |Node |
| | | | |capabilities |
| | | | |supported. |
+-----+-----------------------------+-------+--------+--------------+
|11 | link_capabilities |8.0 | |Capabilities |
| | | | |of this link. |
+-----+-----------------------------+-------+--------+--------------+
|12 | holdtime |8.0 | |Required |
| | | | |holdtime of |
| | | | |the |
| | | | |adjacency, |
| | | | |i.e., for how |
| | | | |long a period |
| | | | |adjacency |
| | | | |should be |
| | | | |kept up |
| | | | |without valid |
| | | | |LIE |
| | | | |reception. |
+-----+-----------------------------+-------+--------+--------------+
|13 | label |8.0 | |Optional, |
| | | | |unsolicited, |
| | | | |downstream |
| | | | |assigned |
| | | | |locally |
| | | | |significant |
| | | | |label value |
| | | | |for the |
| | | | |adjacency. |
+-----+-----------------------------+-------+--------+--------------+
|21 | not_a_ztp_offer |8.0 | |Indicates |
| | | | |that the |
| | | | |level on the |
| | | | |LIE must not |
| | | | |be used to |
| | | | |derive a ZTP |
| | | | |level by the |
| | | | |receiving |
| | | | |node. |
+-----+-----------------------------+-------+--------+--------------+
|22 | you_are_flood_repeater |8.0 | |Indicates to |
| | | | |the |
| | | | |northbound |
| | | | |neighbor that |
| | | | |it should be |
| | | | |reflooding |
| | | | |TIEs received |
| | | | |from this |
| | | | |node to |
| | | | |achieve flood |
| | | | |reduction and |
| | | | |balancing for |
| | | | |northbound |
| | | | |flooding. |
+-----+-----------------------------+-------+--------+--------------+
|23 | you_are_sending_too_quickly |8.0 | |Indicates to |
| | | | |the neighbor |
| | | | |to flood node |
| | | | |TIEs only and |
| | | | |slow down all |
| | | | |other TIEs. |
| | | | |Ignored when |
| | | | |received from |
| | | | |the |
| | | | |southbound |
| | | | |neighbor. |
+-----+-----------------------------+-------+--------+--------------+
|24 | instance_name |8.0 | |Instance name |
| | | | |in case |
| | | | |multiple RIFT |
| | | | |instances are |
| | | | |running on |
| | | | |the same |
| | | | |interface. |
+-----+-----------------------------+-------+--------+--------------+
|35 | fabric_id |8.0 | |It provides |
| | | | |the optional |
| | | | |ID of the |
| | | | |fabric |
| | | | |configured. |
| | | | |This must |
| | | | |match the |
| | | | |information |
| | | | |advertised on |
| | | | |the node |
| | | | |element. |
+-----+---------------------------------+-------+--------+--------------+
Table 24: RIFT LIE Packet10.3.18. RIFTEncodingLinkCapabilities Registry ✨
and here's the link capabilities one 🎀 it starts with the same kind of initial values, so no new mystery here 😌
This registry has the following initial values.
+=====+=========================+=========+==========+==============+
|Value| Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=====+=========================+=========+==========+==============+
|0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-----+-------------------------+---------+----------+--------------+
|1 | bfd | 8.0 | | Indicates |
| | | | | that the |
| | | | | link is |
| | | | | supporting |
| | | | | BFD. |
+-----+-------------------------+---------+----------+--------------+
|2 | ipv4_forwarding_capable | 8.0 | | Indicates |
| | | | | whether the |
| | | | | interface |
| | | | | will |
| | | | | support |
| | | | | IPv4 |
| | | | | forwarding. |
+-----+-------------------------+---------+----------+--------------+
Table 25: Link Capabilities10.3.19. RIFTEncodingLinkIDPair Registry ✨
okay so a LinkID pair describes one of the parallel links between two nodes 🔗💖 like, two nodes can have a bunch of links running between them, and each pair is basically the ID card for one of those links 🪪
this registry has the following initial values 🎀
+=====+============================+=======+========+==============+
|Value| Name |Min. |Max. | Comment |
| | |Schema |Schema | |
| | |Version|Version | |
+=====+============================+=======+========+==============+
|0 | Reserved |8.0 |All | |
| | | |Versions| |
+-----+----------------------------+-------+--------+--------------+
|1 | local_id |8.0 | | Node-wide |
| | | | | unique value |
| | | | | for the |
| | | | | local link. |
+-----+----------------------------+-------+--------+--------------+
|2 | remote_id |8.0 | | Received the |
| | | | | remote link |
| | | | | ID for this |
| | | | | link. |
+-----+----------------------------+-------+--------+--------------+
|10 | platform_interface_index |8.0 | | Describes |
| | | | | the local |
| | | | | interface |
| | | | | index of the |
| | | | | link. |
+-----+----------------------------+-------+--------+--------------+
|11 | platform_interface_name |8.0 | | Describes |
| | | | | the local |
| | | | | interface |
| | | | | name. |
+-----+----------------------------+-------+--------+--------------+
|12 | trusted_outer_security_key |8.0 | | Indicates |
| | | | | whether the |
| | | | | link is |
| | | | | secured, |
| | | | | i.e., |
| | | | | protected by |
| | | | | outer key, |
| | | | | absence of |
| | | | | this element |
| | | | | means no |
| | | | | indication, |
| | | | | undefined |
| | | | | outer key |
| | | | | means not |
| | | | | secured. |
+-----+----------------------------+-------+--------+--------------+
|13 | bfd_up |8.0 | | Indicates |
| | | | | whether the |
| | | | | link is |
| | | | | protected by |
| | | | | an |
| | | | | established |
| | | | | BFD session. |
+-----+----------------------------+-------+--------+--------------+
|14 | address_families |8.0 | | Optional |
| | | | | indication |
| | | | | that address |
| | | | | families are |
| | | | | up on the |
| | | | | interface. |
+-----+----------------------------+-------+--------+--------------+Table 26
10.3.20. RIFTEncodingNeighbor Registry 🧚
this registry has the following initial values 💜
+=======+============+=============+=============+=================+
| Value | Name | Min. Schema | Max. Schema | Comment |
| | | Version | Version | |
+=======+============+=============+=============+=================+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+------------+-------------+-------------+-----------------+
| 1 | originator | 8.0 | | System ID of |
| | | | | the originator. |
+-------+------------+-------------+-------------+-----------------+
| 2 | remote_id | 8.0 | | ID of remote |
| | | | | side of the |
| | | | | link. |
+-------+------------+-------------+-------------+-----------------+Table 27: Neighbor Structure
10.3.21. RIFTEncodingNodeCapabilities Registry 🌸
this registry has the following initial values 🍓
+=====+========================+=========+==========+==============+
|Value| Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=====+========================+=========+==========+==============+
|0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-----+------------------------+---------+----------+--------------+
|1 | protocol_minor_version | 8.0 | | Must |
| | | | | advertise |
| | | | | supported |
| | | | | minor |
| | | | | version |
| | | | | dialect that |
| | | | | way. |
+-----+------------------------+---------+----------+--------------+
|2 | flood_reduction | 8.0 | | Indicates |
| | | | | that node |
| | | | | supports |
| | | | | flood |
| | | | | reduction. |
+-----+------------------------+---------+----------+--------------+
|3 | hierarchy_indications | 8.0 | | Indicates |
| | | | | place in |
| | | | | hierarchy, |
| | | | | i.e., top of |
| | | | | fabric or |
| | | | | leaf only |
| | | | | (in ZTP) or |
| | | | | support for |
| | | | | L2L |
| | | | | procedures. |
+-----+------------------------+---------+----------+--------------+Table 28: Capabilities the Node Supports
10.3.22. RIFTEncodingNodeFlags Registry 🦋
this registry has the following initial values 💅
+=======+==========+=========+==========+===========================+
| Value | Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=======+==========+=========+==========+===========================+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------+---------+----------+---------------------------+
| 1 | overload | 8.0 | | Indicates that node |
| | | | | is in overload; do |
| | | | | not transit traffic |
| | | | | through it. |
+-------+----------+---------+----------+---------------------------+Table 29: Indication Flags of the Node
10.3.23. RIFTEncodingNodeNeighborsTIEElement Registry 🌈
this registry has the following initial values 🫶
+=======+===========+=========+==========+======================+
| Value | Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=======+===========+=========+==========+======================+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+-----------+---------+----------+----------------------+
| 1 | level | 8.0 | | Level of neighbor. |
+-------+-----------+---------+----------+----------------------+
| 3 | cost | 8.0 | | Cost to neighbor. |
| | | | | Ignore anything |
| | | | | equal or larger than |
| | | | | 'infinite_distance' |
| | | | | and equal to |
| | | | | 'invalid_distance'. |
+-------+-----------+---------+----------+----------------------+
| 4 | link_ids | 8.0 | | Carries description |
| | | | | of multiple parallel |
| | | | | links in a TIE. |
+-------+-----------+---------+----------+----------------------+
| 5 | bandwidth | 8.0 | | Total bandwidth to |
| | | | | neighbor as sum of |
| | | | | all parallel links. |
+-------+-----------+---------+----------+----------------------+Table 30: Neighbor of a Node
10.3.24. RIFTEncodingNodeTIEElement Registry 🌸
okay so this registry comes with a bunch of starter values for the node TIE element, and honestly the table is giving everything ✨
+=======+=================+=========+==========+====================+
| Value | Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=======+=================+=========+==========+====================+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+-----------------+---------+----------+--------------------+
| 1 | level | 8.0 | | Level of the |
| | | | | node. |
+-------+-----------------+---------+----------+--------------------+
| 2 | neighbors | 8.0 | | Node's neighbors. |
| | | | | Multiple node |
| | | | | TIEs can carry |
| | | | | disjoint sets of |
| | | | | neighbors. |
+-------+-----------------+---------+----------+--------------------+
| 3 | capabilities | 8.0 | | Capabilities of |
| | | | | the node. |
+-------+-----------------+---------+----------+--------------------+
| 4 | flags | 8.0 | | Flags of the |
| | | | | node. |
+-------+-----------------+---------+----------+--------------------+
| 5 | name | 8.0 | | Optional node |
| | | | | name for easier |
| | | | | operations. |
+-------+-----------------+---------+----------+--------------------+
| 6 | pod | 8.0 | | Pod to which the |
| | | | | node belongs. |
+-------+-----------------+---------+----------+--------------------+
| 7 | startup_time | 8.0 | | Optional startup |
| | | | | time of the node. |
+-------+-----------------+---------+----------+--------------------+
| 10 | miscabled_links | 8.0 | | If any local |
| | | | | links are |
| | | | | miscabled, this |
| | | | | indication is |
| | | | | flooded. |
+-------+-----------------+---------+----------+--------------------+
| 12 | same_plane_tofs | 8.0 | | ToFs in the same |
| | | | | plane. Only |
| | | | | carried by ToF. |
| | | | | Multiple node |
| | | | | TIEs can carry |
| | | | | disjoint sets of |
| | | | | ToFs that must be |
| | | | | joined to form a |
| | | | | single set. |
+-------+-----------------+---------+----------+--------------------+
| 20 | fabric_id | 8.0 | | It provides the |
| | | | | optional ID of |
| | | | | the fabric |
| | | | | configured. |
+-------+-----------------+---------+----------+--------------------+
Table 31: Description of a Node10.3.25. RIFTEncodingPacketContent Registry 💖
this one's the lineup of what can live inside a RIFT packet, starting from its initial values 🫶
+=======+==========+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+==========+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------+---------------------+-------------+---------+
| 1 | lie | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
| 2 | tide | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
| 3 | tire | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
| 4 | tie | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
Table 32: Content of a RIFT Packet10.3.26. RIFTEncodingPacketHeader Registry 🦋
the packet header registry is giving main character energy, bestie. here are its initial values 💅
+=======+===============+=========+==========+===================+
| Value | Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=======+===============+=========+==========+===================+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+---------------+---------+----------+-------------------+
| 1 | major_version | 8.0 | | Major version of |
| | | | | protocol. |
+-------+---------------+---------+----------+-------------------+
| 2 | minor_version | 8.0 | | Minor version of |
| | | | | protocol. |
+-------+---------------+---------+----------+-------------------+
| 3 | sender | 8.0 | | Node sending the |
| | | | | packet; in case |
| | | | | of LIE/TIRE/TIDE |
| | | | | also the |
| | | | | originator of it. |
+-------+---------------+---------+----------+-------------------+
| 4 | level | 8.0 | | Level of the node |
| | | | | sending the |
| | | | | packet, required |
| | | | | on everything |
| | | | | except LIEs. |
| | | | | Lack of presence |
| | | | | on LIEs indicates |
| | | | | undefined_level |
| | | | | and is used in |
| | | | | ZTP procedures. |
+-------+---------------+---------+----------+-------------------+
Table 33: Common RIFT Packet Header10.3.27. RIFTEncodingPrefixAttributes Registry 🧚
prefixes come with a whole bag of attributes, and these are the starter ones, no cap 🌈
+=======+===================+=========+==========+==================+
| Value | Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=======+===================+=========+==========+==================+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+-------------------+---------+----------+------------------+
| 2 | metric | 8.0 | | Distance of the |
| | | | | prefix. |
+-------+-------------------+---------+----------+------------------+
| 3 | tags | 8.0 | | Generic |
| | | | | unordered set |
| | | | | of route tags, |
| | | | | can be |
| | | | | redistributed |
| | | | | to other |
| | | | | protocols or |
| | | | | used within the |
| | | | | context of real |
| | | | | time analytics. |
+-------+-------------------+---------+----------+------------------+
| 4 | monotonic_clock | 8.0 | | Monotonic clock |
| | | | | for mobile |
| | | | | addresses. |
+-------+-------------------+---------+----------+------------------+
| 6 | loopback | 8.0 | | Indicates if |
| | | | | the prefix is a |
| | | | | node loopback. |
+-------+-------------------+---------+----------+------------------+
| 7 | directly_attached | 8.0 | | Indicates that |
| | | | | the prefix is |
| | | | | directly |
| | | | | attached. |
+-------+-------------------+---------+----------+------------------+
| 10 | from_link | 8.0 | | Link to which |
| | | | | the address |
| | | | | belongs to. |
+-------+-------------------+---------+----------+------------------+
| 12 | label | 8.0 | | Optional, per- |
| | | | | prefix |
| | | | | significant |
| | | | | label. |
+-------+-------------------+---------+----------+------------------+
Table 34: Attributes of a Prefix10.3.28. RIFTEncodingPrefixTIEElement Registry 👑
last one in this batch: the prefix TIE element registry and its initial values, iconic ✨
+=======+==========+=============+=============+================+
| Value | Name | Min. Schema | Max. Schema | Comment |
| | | Version | Version | |
+=======+==========+=============+=============+================+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------+-------------+-------------+----------------+
| 1 | prefixes | 8.0 | | Prefixes with |
| | | | | the associated |
| | | | | attributes. |
+-------+----------+-------------+-------------+----------------+
Table 35: TIE Carrying Prefixes10.3.29. RIFTEncodingProtocolPacket Registry 🎀
okay so this registry comes with these starter values 💅
+=======+==========+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+==========+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------+---------------------+-------------+---------+
| 1 | header | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
| 2 | content | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
Table 36: RIFT Packet Structure10.3.30. RIFTEncodingTIDEPacket Registry 🦋
this one also has starter values, and they're giving organized 📋
+=======+=============+=============+=============+===============+
| Value | Name | Min. Schema | Max. Schema | Comment |
| | | Version | Version | |
+=======+=============+=============+=============+===============+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+-------------+-------------+-------------+---------------+
| 1 | start_range | 8.0 | | First TIE |
| | | | | header in the |
| | | | | TIDE packet. |
+-------+-------------+-------------+-------------+---------------+
| 2 | end_range | 8.0 | | Last TIE |
| | | | | header in the |
| | | | | TIDE packet. |
+-------+-------------+-------------+-------------+---------------+
| 3 | headers | 8.0 | | _sorted_ list |
| | | | | of headers. |
+-------+-------------+-------------+-------------+---------------+
Table 37: TIDE with Sorted TIE Headers10.3.31. RIFTEncodingTIEElement Registry ✨
starter values again, and this table is literally a whole vibe 🌈
+=====+========================+=======+========+===================+
|Value|Name |Min. |Max. |Comment |
| | |Schema |Schema | |
| | |Version|Version | |
+=====+========================+=======+========+===================+
|0 |Reserved |8.0 |All | |
| | | |Versions| |
+-----+------------------------+-------+--------+-------------------+
|1 |node |8.0 | |Used in case of |
| | | | |enum |
| | | | |common.tietypetype.|
| | | | |nodetietype. |
+-----+------------------------+-------+--------+-------------------+
|2 |prefixes |8.0 | |Used in case of |
| | | | |enum |
| | | | |common.tietypetype.|
| | | | |prefixtietype. |
+-----+------------------------+-------+--------+-------------------+
|3 |positive_disaggregation_|8.0 | |Positive prefixes |
| |prefixes | | |(always |
| | | | |southbound). |
+-----+------------------------+-------+--------+-------------------+
|5 |negative_disaggregation_|8.0 | |Transitive, |
| |prefixes | | |negative prefixes |
| | | | |(always southbound)|
+-----+------------------------+-------+--------+-------------------+
|6 |external_prefixes |8.0 | |Externally |
| | | | |reimported |
| | | | |prefixes. |
+-----+------------------------+-------+--------+-------------------+
|7 |positive_external_ |8.0 | |Positive external |
| |disaggregation_prefixes | | |disaggregated |
| | | | |prefixes |
| | | | |(always |
| | | | |southbound). |
+-----+------------------------+-------+--------+-------------------+
|9 |keyvalues |8.0 | |Key-value |
| | | | |store elements. |
+-----+------------------------+-------+--------+-------------------+
Table 38: Single Element in a TIE10.3.32. RIFTEncodingTIEHeader Registry 💜
this is the header of a TIE, and it's got its own starter set 🫶
+=======+======================+=========+==========+==============+
| Value | Name | Min. | Max. | Comment |
| | | Schema | Schema | |
| | | Version | Version | |
+=======+======================+=========+==========+==============+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------------------+---------+----------+--------------+
| 2 | tieid | 8.0 | | ID of TIE. |
+-------+----------------------+---------+----------+--------------+
| 3 | seq_nr | 8.0 | | Sequence |
| | | | | number of |
| | | | | TIE. |
+-------+----------------------+---------+----------+--------------+
| 10 | origination_time | 8.0 | | Absolute |
| | | | | timestamp |
| | | | | when TIE was |
| | | | | generated. |
+-------+----------------------+---------+----------+--------------+
| 12 | origination_lifetime | 8.0 | | Original |
| | | | | lifetime |
| | | | | when TIE was |
| | | | | generated. |
+-------+----------------------+---------+----------+--------------+
Table 39: Header of a TIE10.3.33. RIFTEncodingTIEHeaderWithLifeTime Registry 🌸
same deal here, starter values, no notes needed 💅
+=======+====================+=============+==========+===========+
| Value | Name | Min. Schema | Max. | Comment |
| | | Version | Schema | |
| | | | Version | |
+=======+====================+=============+==========+===========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+--------------------+-------------+----------+-----------+
| 1 | header | 8.0 | | |
+-------+--------------------+-------------+----------+-----------+
| 2 | remaining_lifetime | 8.0 | | Remaining |
| | | | | lifetime. |
+-------+--------------------+-------------+----------+-----------+
Table 40: Header of a TIE as Described in TIRE/TIDE10.3.34. RIFTEncodingTIEID Registry 🧚
the unique ID of a TIE gets its starter values too ✨
+=======+============+=============+=============+============+
| Value | Name | Min. Schema | Max. Schema | Comment |
| | | Version | Version | |
+=======+============+=============+=============+============+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+------------+-------------+-------------+------------+
| 1 | direction | 8.0 | | Direction |
| | | | | of TIE. |
+-------+------------+-------------+-------------+------------+
| 2 | originator | 8.0 | | Indicates |
| | | | | originator |
| | | | | of TIE. |
+-------+------------+-------------+-------------+------------+
| 3 | tietype | 8.0 | | Type of |
| | | | | TIE. |
+-------+------------+-------------+-------------+------------+
| 4 | tie_nr | 8.0 | | Number of |
| | | | | TIE. |
+-------+------------+-------------+-------------+------------+
Table 41: Unique ID of a TIE10.3.35. RIFTEncodingTIEPacket Registry 🌈
the TIE packet has its own starter table, slay 👑
+=======+==========+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+==========+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------+---------------------+-------------+---------+
| 1 | header | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
| 2 | element | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
Table 42: TIE Packet10.3.36. RIFTEncodingTIREPacket Registry 🍓
last registry of this batch, and it also comes with starter values 🦋
+=======+==========+=====================+=============+=========+
| Value | Name | Min. Schema Version | Max. Schema | Comment |
| | | | Version | |
+=======+==========+=====================+=============+=========+
| 0 | Reserved | 8.0 | All | |
| | | | Versions | |
+-------+----------+---------------------+-------------+---------+
| 1 | headers | 8.0 | | |
+-------+----------+---------------------+-------------+---------+
Table 43: TIRE Packet11. References 📚
11.1. Normative References ✨
okay bestie, this is where the receipts live 📚 these are the docs the spec totally leans on, so they're required reading 🫶
[EUI64] IEEE, "Guidelines for Use of Extended Unique Identifier (EUI), Organizationally Unique Identifier (OUI), and Company ID (CID)", https://standards-support.ieee.org/hc/en-us/articles/4888705676564-Guidelines-for-Use-of-Extended-Unique-Identifier-EUI-Organizationally-Unique-Identifier-OUI-and-Company-ID-CID.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, https://www.rfc-editor.org/info/rfc2119.
[RFC2365] Meyer, D., "Administratively Scoped IP Multicast", BCP 23, RFC 2365, DOI 10.17487/RFC2365, July 1998, https://www.rfc-editor.org/info/rfc2365.
[RFC4291] Hinden, R. and S. Deering, "IP Version 6 Addressing Architecture", RFC 4291, DOI 10.17487/RFC4291, February 2006, https://www.rfc-editor.org/info/rfc4291.
[RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C. Pignataro, "The Generalized TTL Security Mechanism (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007, https://www.rfc-editor.org/info/rfc5082.
[RFC5120] Przygienda, T., Shen, N., and N. Sheth, "M-ISIS: Multi Topology (MT) Routing in Intermediate System to Intermediate Systems (IS-ISs)", RFC 5120, DOI 10.17487/RFC5120, February 2008, https://www.rfc-editor.org/info/rfc5120.
[RFC5709] Bhatia, M., Manral, V., Fanto, M., White, R., Barnes, M., Li, T., and R. Atkinson, "OSPFv2 HMAC-SHA Cryptographic Authentication", RFC 5709, DOI 10.17487/RFC5709, October 2009, https://www.rfc-editor.org/info/rfc5709.
[RFC5881] Katz, D. and D. Ward, "Bidirectional Forwarding Detection (BFD) for IPv4 and IPv6 (Single Hop)", RFC 5881, DOI 10.17487/RFC5881, June 2010, https://www.rfc-editor.org/info/rfc5881.
[RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithms Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, https://www.rfc-editor.org/info/rfc5905.
[RFC7987] Ginsberg, L., Wells, P., Decraene, B., Przygienda, T., and H. Gredler, "IS-IS Minimum Remaining Lifetime", RFC 7987, DOI 10.17487/RFC7987, October 2016, https://www.rfc-editor.org/info/rfc7987.
[RFC8017] Moriarty, K., Ed., Kaliski, B., Jonsson, J., and A. Rusch, "PKCS #1: RSA Cryptography Specifications Version 2.2", RFC 8017, DOI 10.17487/RFC8017, November 2016, https://www.rfc-editor.org/info/rfc8017.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, https://www.rfc-editor.org/info/rfc8174.
[RFC8200] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", STD 86, RFC 8200, DOI 10.17487/RFC8200, July 2017, https://www.rfc-editor.org/info/rfc8200.
[RFC8202] Ginsberg, L., Previdi, S., and W. Henderickx, "IS-IS Multi-Instance", RFC 8202, DOI 10.17487/RFC8202, June 2017, https://www.rfc-editor.org/info/rfc8202.
[RFC8505] Thubert, P., Ed., Nordmark, E., Chakrabarti, S., and C. Perkins, "Registration Extensions for IPv6 over Low-Power Wireless Personal Area Network (6LoWPAN) Neighbor Discovery", RFC 8505, DOI 10.17487/RFC8505, November 2018, https://www.rfc-editor.org/info/rfc8505.
[RFC9300] Farinacci, D., Fuller, V., Meyer, D., Lewis, D., and A. Cabellos, Ed., "The Locator/ID Separation Protocol (LISP)", RFC 9300, DOI 10.17487/RFC9300, October 2022, https://www.rfc-editor.org/info/rfc9300.
[RFC9301] Farinacci, D., Maino, F., Fuller, V., and A. Cabellos, Ed., "Locator/ID Separation Protocol (LISP) Control Plane", RFC 9301, DOI 10.17487/RFC9301, October 2022, https://www.rfc-editor.org/info/rfc9301.
[SHA-2] NIST, "Secure Hash Standard (SHS)", FIPS PUB 180-4, DOI 10.6028/NIST.FIPS.180-4, July 2015, https://csrc.nist.gov/pubs/fips/180-4/upd1/final.
[thrift] Apache Software Foundation, "Apache Thrift Documentation", https://thrift.apache.org/docs/.
11.2. Informative References 🌸
these ones are the extra reading 📖 totally worth a look if you're curious about the backstory 🦋
[CLOS] Yuan, X., "On Nonblocking Folded-Clos Networks in Computer Communication Environments", 2011 IEEE International Parallel & Distributed Processing Symposium, DOI 10.1109/IPDPS.2011.27, May 2011, https://ieeexplore.ieee.org/document/6012836.
[DayOne] Aelmans, M., Vandezande, O., Rijsman, B., Head, J., Graf, C., Alberro, L., Mali, H., and O. Steudler, "Day One: Routing in Fat Trees (RIFT)", Juniper Network Books, ISBN 978-1-7363160-0-9, December 2020.
[DIJKSTRA] Dijkstra, E. W., "A Note on Two Problems in Connexion with Graphs", Numerische Mathematik, vol. 1, pp. 269-271, DOI 10.1007/BF01386390, December 1959, https://link.springer.com/article/10.1007/BF01386390.
[DYNAMO] De Candia, G., Hastorun, D., Jampani, M., Kakulpati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., and W. Vogels, "Dynamo: amazon's highly available key-value store", ACM SIGOPS Operating Systems Review, vol. 41, no. 6, pp. 205-220, DOI 10.1145/1323293.1294281, October 2007, https://dl.acm.org/doi/10.1145/1323293.1294281.
[EPPSTEIN] Eppstein, D., "Finding the k Shortest Paths", March 1997, https://ics.uci.edu/~eppstein/pubs/Epp-SJC-98.pdf.
[FATTREE] Leiserson, C. E., "Fat-Trees: Universal Networks for Hardware-Efficient Supercomputing", IEEE Transactions on Computers, vol. C-34, no. 10, pp. 892-901, DOI 10.1109/TC.1985.6312192, October 1985, https://ieeexplore.ieee.org/document/6312192.
[IEEEstd1588] IEEE, "IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems", IEEE Std 1588-2008, DOI 10.1109/IEEESTD.2008.4579760, July 2008, https://ieeexplore.ieee.org/document/4579760/.
[IEEEstd8021AS] IEEE, "IEEE Standard for Local and Metropolitan Area Networks - Timing and Synchronization for Time-Sensitive Applications in Bridged Local Area Networks", IEEE Std 802.1AS-2011, DOI 10.1109/IEEESTD.2011.5741898, March 2011, https://ieeexplore.ieee.org/document/5741898/.
[RFC0826] Plummer, D., "An Ethernet Address Resolution Protocol: Or Converting Network Protocol Addresses to 48.bit Ethernet Address for Transmission on Ethernet Hardware", STD 37, RFC 826, DOI 10.17487/RFC0826, November 1982, https://www.rfc-editor.org/info/rfc826.
[RFC1982] Elz, R. and R. Bush, "Serial Number Arithmetic", RFC 1982, DOI 10.17487/RFC1982, August 1996, https://www.rfc-editor.org/info/rfc1982.
[RFC2104] Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed-Hashing for Message Authentication", RFC 2104, DOI 10.17487/RFC2104, February 1997, https://www.rfc-editor.org/info/rfc2104.
[RFC2131] Droms, R., "Dynamic Host Configuration Protocol", RFC 2131, DOI 10.17487/RFC2131, March 1997, https://www.rfc-editor.org/info/rfc2131.
[RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, DOI 10.17487/RFC2474, December 1998, https://www.rfc-editor.org/info/rfc2474.
[RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and Multicast Next-Hop Selection", RFC 2991, DOI 10.17487/RFC2991, November 2000, https://www.rfc-editor.org/info/rfc2991.
[RFC4086] Eastlake 3rd, D., Schiller, J., and S. Crocker, "Randomness Requirements for Security", BCP 106, RFC 4086, DOI 10.17487/RFC4086, June 2005, https://www.rfc-editor.org/info/rfc4086.
[RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman, "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861, DOI 10.17487/RFC4861, September 2007, https://www.rfc-editor.org/info/rfc4861.
[RFC4862] Thomson, S., Narten, T., and T. Jinmei, "IPv6 Stateless Address Autoconfiguration", RFC 4862, DOI 10.17487/RFC4862, September 2007, https://www.rfc-editor.org/info/rfc4862.
[RFC5837] Atlas, A., Ed., Bonica, R., Ed., Pignataro, C., Ed., Shen, N., and JR. Rivers, "Extending ICMP for Interface and Next-Hop Identification", RFC 5837, DOI 10.17487/RFC5837, April 2010, https://www.rfc-editor.org/info/rfc5837.
[RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010, https://www.rfc-editor.org/info/rfc5880.
[RFC6550] Winter, T., Ed., Thubert, P., Ed., Brandt, A., Hui, J., Kelsey, R., Levis, P., Pister, K., Struik, R., Vasseur, JP., and R. Alexander, "RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks", RFC 6550, DOI 10.17487/RFC6550, March 2012, https://www.rfc-editor.org/info/rfc6550.
[RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, https://www.rfc-editor.org/info/rfc8126.
[RFC8415] Mrugalski, T., Siodelski, M., Volz, B., Yourtchenko, A., Richardson, M., Jiang, S., Lemon, T., and T. Winters, "Dynamic Host Configuration Protocol for IPv6 (DHCPv6)", RFC 8415, DOI 10.17487/RFC8415, November 2018, https://www.rfc-editor.org/info/rfc8415.
[RFC9696] Wei, Y., Ed., Zhang, Z., Afanasiev, D., Thubert, P., and T. Przygienda, "Routing in Fat Trees (RIFT) Applicability and Operational Considerations", RFC 9696, DOI 10.17487/RFC9696, April 2025, https://www.rfc-editor.org/info/rfc9696.
[VAHDAT08] Al-Fares, M., Loukissas, A., and A. Vahdat, "A Scalable, Commodity Data Center Network Architecture", ACM SIGCOMM Computer Communication Review, vol. 38, no. 4, pp. 63-74, DOI 10.1145/1402946.1402967, August 2008, https://dl.acm.org/doi/10.1145/1402946.1402967.
[VFR] Giotsas, V. and S. Zhou, "Valley-free violation in Internet routing - Analysis based on BGP Community data", 2012 IEEE International Conference on Communications (ICC), DOI 10.1109/ICC.2012.6363987, June 2012, https://ieeexplore.ieee.org/document/6363987.
Appendix A. Sequence Number Binary Arithmetic 🧮✨
Okay bestie, this appendix is all about a sneaky variant of sequence number arithmetic that's related to [RFC1982] 🦋. It gets explained over two's complement arithmetic, which is honestly so easy to implement 💅.
Assuming straight two's complement subtraction on the bit width of the sequence numbers, the matching >: and =: relations work like this:
U_1,U_2are 12-bits aligned unsigned version numbers 🔢D_fis( U_1 - U_2 )interpreted as two's complement signed 12-bitsD_bis( U_2 - U_1 )interpreted as two's complement signed 12-bitsU_1 >: U_2IIFD_f > 0andD_b < 0🙌U_1 =: U_2IIFD_f = 0💖
The >: relationship is anti-symmetric but not transitive, so no cap, it gets a little messy 🌀. Observe that this leaves >: undefined for numbers that sit at the maximum two's complement distance from each other. For example, ( 0 and 0x800 ) is undefined in the 12-bits case because D_f and D_b are both -0x7ff 🙈.
Here's a simple example for 3-bit arithmetic. The first table shows the D_f/D_b values, and the second one shows how U_1 relates to U_2 🎀.
+=========+=====+=====+=====+=====+=====+=====+=====+=====+
| U2 / U1 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+=========+=====+=====+=====+=====+=====+=====+=====+=====+
| 0 | +/+ | +/- | +/- | +/- | -/- | -/+ | -/+ | -/+ |
+---------+-----+-----+-----+-----+-----+-----+-----+-----+
| 1 | -/+ | +/+ | +/- | +/- | +/- | -/- | -/+ | -/+ |
+---------+-----+-----+-----+-----+-----+-----+-----+-----+
| 2 | -/+ | -/+ | +/+ | +/- | +/- | +/- | -/- | -/+ |
+---------+-----+-----+-----+-----+-----+-----+-----+-----+
| 3 | -/+ | -/+ | -/+ | +/+ | +/- | +/- | +/- | -/- |
+---------+-----+-----+-----+-----+-----+-----+-----+-----+
| 4 | -/- | -/+ | -/+ | -/+ | +/+ | +/- | +/- | +/- |
+---------+-----+-----+-----+-----+-----+-----+-----+-----+
| 5 | +/- | -/- | -/+ | -/+ | -/+ | +/+ | +/- | +/- |
+---------+-----+-----+-----+-----+-----+-----+-----+-----+
| 6 | +/- | +/- | -/- | -/+ | -/+ | -/+ | +/+ | +/- |
+---------+-----+-----+-----+-----+-----+-----+-----+-----+
| 7 | +/- | +/- | +/- | -/- | -/+ | -/+ | -/+ | +/+ |
+---------+-----+-----+-----+-----+-----+-----+-----+-----+Table 44
+=========+===+===+===+===+===+===+===+===+
| U2 / U1 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+=========+===+===+===+===+===+===+===+===+
| 0 | = | > | > | > | ? | < | < | < |
+---------+---+---+---+---+---+---+---+---+
| 1 | < | = | > | > | > | ? | < | < |
+---------+---+---+---+---+---+---+---+---+
| 2 | < | < | = | > | > | > | ? | < |
+---------+---+---+---+---+---+---+---+---+
| 3 | < | < | < | = | > | > | > | ? |
+---------+---+---+---+---+---+---+---+---+
| 4 | ? | < | < | < | = | > | > | > |
+---------+---+---+---+---+---+---+---+---+
| 5 | > | ? | < | < | < | = | > | > |
+---------+---+---+---+---+---+---+---+---+
| 6 | > | > | ? | < | < | < | = | > |
+---------+---+---+---+---+---+---+---+---+
| 7 | > | > | > | ? | < | < | < | = |
+---------+---+---+---+---+---+---+---+---+Table 45
Appendix B. Examples 🌸
B.1. Normal Operation 🏠
Let's look at the topology picture first, because it's giving everything 👑.
^ N +--------+ +--------+
Level 2 | |ToF 21| |ToF 22|
W <-*-> E ++-+--+-++ ++-+--+-++
| | | | | | | | |
S v P111/2 P121/2 | | | |
MHP MHP | | | |
^ ^ ^ ^ | | | |
| | | | | | | |
+--------------+ | +-----------+ | | | +---------------+
| | | | | | | |
| +-----------------------------+ | | ^
| | | | | | | All
0/0 0/0 0/0 +-----------------------------+ TIEs
v v v | | | | |
| | +-+ +<-0/0----------+ | |
| | | | | | | |
+-+----++ Optional +-+----++ ++----+-+ ++-----++
Level 1 | | E/W Link | | | | | |
|Spin111+----------+Spin112| |Spin121| |Spin122|
+-+---+-+ ++----+-+ +-+---+-+ ++---+--+
| | | | | | | |
| +---0/0--->-----+ 0/0 | +----------------+ |
0/0 | | | | | | |
| +---<-0/0-----+ | v | +--------------+ | |
v | | | | | | |
+-+---+-+ +--+--+-+ +-+---+-+ +---+-+-+
Level 0 | | L2L | | | | | |
|Leaf111+~~~~~~~~~~+Leaf112| |Leaf121| |Leaf122|
+-+-----+ +-+---+-+ +--+--+-+ +-+-----+
+ + \ / + +
Prefix111 Prefix112 \ / Prefix121 Prefix122
Multi-homed
Prefix
+---------- PoD 1 ---------+ +---------- PoD 2 ---------+Figure 35: Normal Case Topology
This section walks through RIFT deployment in the example topology from Figure 35 with no node or link failures 🧚. To keep things simple, it skips flooding reduction and shortens some node names so everything fits in the picture 🫶.
First up, these bidirectional adjacencies get established:
- ToF 21 (PoD 0) to Spine 111, Spine 112, Spine 121, and Spine 122
- ToF 22 (PoD 0) to Spine 111, Spine 112, Spine 121, and Spine 122
- Spine 111 to Leaf 111 and Leaf 112
- Spine 112 to Leaf 111 and Leaf 112
- Spine 121 to Leaf 121 and Leaf 122
- Spine 122 to Leaf 121 and Leaf 122
Leaf 111 and Leaf 112 originate N-TIEs for Prefix 111 and Prefix 112 (respectively) to both Spine 111 and Spine 112. Leaf 112 also originates an N-TIE for the multihomed prefix 🍓. Then Spine 111 and Spine 112 originate their own N-TIEs, and they flood the N-TIEs they got from Leaf 111 and Leaf 112 up to both ToF 21 and ToF 22 💅.
Leaf 121 and Leaf 122 do the same thing with the other side: they originate North TIEs for Prefix 121 and Prefix 122 (respectively) to Spine 121 and Spine 122. Leaf 121 also originates a North TIE for the multihomed prefix ✨. Spine 121 and Spine 122 then originate their own North TIEs and flood the North TIEs from Leaf 121 and Leaf 122 to both ToF 21 and ToF 22 🦋.
Spines only hold the North TIEs of level 0 for their PoD, and leaves only hold their own North TIEs. At this point though, ToF 21 and ToF 22 (plus any northbound connected controllers) have the complete network topology 🌈.
ToF 21 and ToF 22 then originate and flood South TIEs. These contain the established adjacencies and a default IP route to all spines 💜. Spine 111, Spine 112, Spine 121, and Spine 122 reflect all South Node TIEs they get from ToF 21 over to ToF 22, and the other way around too. South TIEs never get re-propagated southbound, periodt 🙅♀️.
Next, Spine 111 and Spine 112 each originate South TIEs with a default IP route toward Leaf 111 and Leaf 112. Spine 121 and Spine 122 do the same toward Leaf 121 and Leaf 122 🎀.
And just like that, IP connectivity across the maximum number of viable paths is set up for all leaves 🎉. Routing info is kept to the bare minimum needed for normal operation and redundancy, which is so clean 👑.
B.2. Leaf Link Failure 💔
| | | |
+-+---+-+ +-+---+-+
| | | |
|Spin111| |Spin112|
+-+---+-+ ++----+-+
| | | |
| +---------------+ X
| | | X
| +-------------+ | X
| | | |
+-+---+-+ +--+--+-+
| | | |
|Leaf111| |Leaf112|
+-------+ +-------+
+ +
Prefix111 Prefix112Figure 36: Single Leaf Link Failure
Okay, now the drama 😭. Say the link between Spine 112 and Leaf 112 goes down. Both nodes originate new Node TIEs that list their connected adjacencies, minus the one that just died 💀. Leaf 112 sends a North Node TIE to Spine 111. Spine 112 sends a North Node TIE to ToF 21 and ToF 22, plus a new South Node TIE to Leaf 111, which gets reflected to Spine 111 🔁. Then the SPF gets recomputed, and Spine 112 is no longer in the forwarding path for Prefix 112 🙃.
Spine 111 also disaggregates Prefix 112 by sending a new South Prefix TIE to Leaf 111 and Leaf 112 🧠. Disaggregation gets its own deeper dive in the next section, but it's worth flagging here because it shows how RIFT protects against traffic loss 🛟. Picture this: Leaf 111 hasn't received the more specific (disaggregated) route from Spine 111 yet. Traffic from Leaf 111 toward Prefix 112 might still use Spine 112's default route, which sends it up through ToF 21 and ToF 22 and back down via Spine 111 🌀. Is that suboptimal? Yes, bestie. Is it temporary? Also yes. And it's still better than dropping traffic, so we'll take it 🫶.
B.3. Partitioned Fabric 🧩
+--------+ +--------+
Level 2 |ToF 21| |ToF 22|
++-+--+-++ ++-+--+-++
| | | | | | | |
| | | | | | | 0/0
| | | | | | | |
| | | | | | | |
+--------------+ | +--- XXXXXX + | | | +---------------+
| | | | | | | |
| +-----------------------------+ | | |
0/0 | | | | | | |
| 0/0 0/0 +- XXXXXXXXXXXXXXXXXXXXXXXXX -+ |
| 1.1/16 | | | | | |
| | +-+ +-0/0-----------+ | |
| | | 1.1./16 | | | |
+-+----++ +-+-----+ ++-----0/0 ++----0/0
Level 1 | | | | | 1.1/16 | 1.1/16
|Spin111| |Spin112| |Spin121| |Spin122|
+-+---+-+ ++----+-+ +-+---+-+ ++---+--+
| | | | | | | |
| +---------------+ | | +----------------+ |
| | | | | | | |
| +-------------+ | | | +--------------+ | |
| | | | | | | |
+-+---+-+ +--+--+-+ +-+---+-+ +---+-+-+
Level 3 | | | | | | | |
|Leaf111| |Leaf112| |Leaf121| |Leaf122|
+-+-----+ ++------+ +-----+-+ +-+-----+
+ + + +
Prefix111 Prefix112 Prefix121 Prefix122
1.1/16
Figure 37: Fabric PartitionOkay, this one is the spicy scenario 🌶️🔥. Figure 37 shows ToF 21 getting completely cut off from Prefix 121 because of a double link failure 💀. If the fabric only had default routes, that would mean 50% of the traffic from Leaf 111 and Leaf 112 heading toward Prefix 121 just gets dropped. Not it. 🙅♀️
Here's how the fix works 💅. ToF 21's South TIEs get reflected from Spine 111 and Spine 112 over to ToF 22. So once ToF 22 finds out that Prefix 121 is unreachable from ToF 21, it starts disaggregating Prefix 121. It advertises a more specific route (1.1/16) to all spines, along with the default IP prefix route. ToF 21 still only sends the default route, so no change there. The end result is that Spine 111 and Spine 112 use that more specific route to reach Prefix 121 through ToF 22. Everything else keeps using the default IP prefix route toward both ToF 21 and ToF 22, so it's giving stability ✨.
The more specific route for Prefix 121 doesn't need to go any further south to the leaves, because they don't get anything useful out of it 🙃. Spine 111 and Spine 112 only have to reflect the new South Node TIEs they get from ToF 22 up to ToF 21. So only the nodes that actually need the info get it, and the failure stays contained to the partitioned level 🦋. The rest of the fabric doesn't have to deal with extra flooding or recomputing the topology, which is honestly so iconic.
To wrap up this example, here are the sets ToF 22 computes using the notation from Section 6.5 🧮:
|R = Prefix 111, Prefix 112, Prefix 121, Prefix 122
|H (for r=Prefix 111) = Spine 111, Spine 112
|H (for r=Prefix 112) = Spine 111, Spine 112
|H (for r=Prefix 121) = Spine 121, Spine 122
|H (for r=Prefix 122) = Spine 121, Spine 122
|A (for ToF 21) = Spine 111, Spine 112Since |H (for r=Prefix 121) and |H (for r=Prefix 122) are both disjoint from |A (for ToF 21), ToF 22 originates a South TIE carrying Prefix 121 and Prefix 122, and that gets flooded to all spines 🎀✨.
B.4. Northbound Partitioned Router and Optional East-West Links 🌈
+ + +
X N1 | N2 | N3
X | |
+--+----+ +--+----+ +--+-----+
| |0/0> <0/0| |0/0> <0/0| |
| A01 +----------+ A02 +----------+ A03 | Level 1
++-+-+--+ ++--+--++ +---+-+-++
| | | | | | | | |
| | +----------------------------------+ | | |
| | | | | | | | |
| +-------------+ | | | +--------------+ |
| | | | | | | | |
| +----------------+ | +-----------------+ |
| | | | | | | | |
| | +------------------------------------+ | |
| | | | | | | | |
++-+-+--+ | +---+---+ | +-+---+-++
| | +-+ +-+ | |
| L01 | | L02 | | L03 | Level 0
+-------+ +-------+ +--------+
Figure 38: North Partitioned RouterFigure 38 is a look at a fabric where level 1 is connected horizontally, and A01 lost its only northbound adjacency 😬. Using the N-SPF rules in Section 6.4.1, A01 works out its northbound reachability by going through the link from A01 to A02. But A02 does not use that link during N-SPF, so A01 ends up using the horizontal link to advertise its default route and do unidirectional routing 🔀.
Now, if A02 also loses its only northbound adjacency (N2), things get messier 🫠. A01 no longer has northbound reachability, but it does still receive A03's northbound adjacencies in South Node TIEs that get reflected by the nodes south of it. Because of that, A01 stops advertising its default route, following Section 6.3.8. Plot twist, but it's the correct move 💜.
Acknowledgments ✨
A new routing protocol this complex is never just one person's baby 👶💕. It takes a whole village, and the author list shows that. Plus, lots of other people gave input and went through the spec with a fine-tooth comb, drawing on their experience designing, implementing, and using protocols in IP fabrics. This section is only a small attempt to record all of that 🫶.
Huge thanks to Naiming Shen for the early chats about using IGPs for routing in Clos-related topologies 🌸. Russ White gets a special shout-out for the key conversation on epistemology, which connected current asynchronous distributed systems theory to the modern protocol design in this document 🧠✨. Adrian Farrel, Joel Halpern, Jeffrey Zhang, Krzysztof Szarkowicz, Nagendra Kumar, Melchior Aelmans, Kaushal Tank, Will Jones, Moin Ahmed, Zheng (Sandy) Zhang, and Donald Eastlake gave thoughtful comments that made the document easier to read and caught a bunch of corners where the light hadn't reached yet 🔦. Kris Price was the first to bring up single router, single arm default considerations 🙌. Jeff Tantsura shared some early thoughts on BFD interactions, and Jeff Haas corrected several misconceptions about BFD's finer points and helped improve the security section around leaf considerations 🛡️💅. Artur Makutunowicz suggested lots of possible improvements and was a sounding board for the modern implementation techniques RIFT is exploring 🦋. Barak Gafni clearly formalized the problem of a partitioned spine and fallen leaves for the first time, on a (very clean) napkin in Singapore 🍓. That napkin led to the super important parts of the spec about multiple ToF planes and negative disaggregation 👑. Igor Gashinsky and others shared a ton of thoughts on problems they ran into while designing and operating large-scale data center fabrics 🏢. And Xu Benchong found a delicate error in the flooding procedures and a schema datatype size mismatch 🔍💖.
Too many people to name also reviewed the document from all directions across the IETF 🥹. Some pointed out critical defects, some asked for things that an earlier reviewer had already removed as objectionable or superfluous, and many said the document was either too crowded with the obvious or left cryptic concepts everywhere with no introduction. Honestly, the editors did their best to find a balance. The document guides the reader through Section 2 into a spec tight enough for interoperable implementations, while also giving enough operational context about routable IP fabrics to keep a concise, shared vocabulary for the unfamiliar concepts the protocol relies on 📚✨. The goal was to avoid ending up with Aesop's donkey 🫏 (sorry, had to). So the result might not feel perfect to everyone, but it should be more than good enough for everyone who ends up using it 💜.
Last but definitely not least, Alvaro Retana, John Scudder, Andrew Alston, and Jim Guichard guided the whole thing as ADs. They asked lots of necessary procedural and technical questions, which not only made the content better but also laid out the path to publication 🛤️🎀. And Roman Danyliw goes very last, but he's a true legend for his painstakingly detailed review and for improving the security aspects of the spec 🔐👑.
Contributors 💖
This work was produced by a long list of people who are all major contributors, whether or not their names made it onto the limited author list 🫶.
Tony Przygienda, Ed.
Juniper
Jordan Head, Ed.
Juniper
Alankar Sharma
Hudson River Trading
Pascal Thubert
Cisco
Bruno Rijsman
Individual
Dmitry Afanasiev
Individual
Don Fedyk
LabN
Alia Atlas
Individual
John Drake
Individual
Ilya Vershkov
NvidiaAuthors' Addresses 🦋
Tony Przygienda (editor)
Juniper Networks
1137 Innovation Way
Sunnyvale, CA 94089
United States of America
Email: prz@juniper.net
Jordan Head (editor)
Juniper Networks
1137 Innovation Way
Sunnyvale, CA 94089
United States of America
Email: jhead@juniper.net
Alankar Sharma
Hudson River Trading
United States of America
Email: as3957@gmail.com
Pascal Thubert
Individual
France
Email: pascal.thubert@gmail.com
Bruno Rijsman
Individual
Email: brunorijsman@gmail.com
Dmitry Afanasiev
Yandex
Email: fl0w@yandex-team.ru📦 where this page comes from
- ietf: rfc9692 2025-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms