RFC 2991

Multipath Issues in Unicast and Multicast Next-Hop Selection

📅 published November 2000, from ietf

RFC 2991 💖 Multipath Issues in Unicast and Multicast Next-Hop Selection ✨

Network Working Group 🎀 Authors: D. Thaler (Microsoft) and C. Hopps (NextHop Technologies). Category: Informational 🌸 Date: November 2000 🧚

Status of this Memo 🌈

okay so this memo is just providing information for the Internet community 🫶 it is not a spec for an Internet standard of any kind. distribution is unlimited, so share it with everyone bestie 💅

Copyright (C) The Internet Society (2000). All Rights Reserved.

Abstract 🦋

ok so you know how routing protocols like OSPF and ISIS let you do "Equal-Cost Multipath" (ECMP) routing? 🔥 some router implementations also let you do equal-cost multipath with RIP and other routing protocols. the vibe is: a forwarder can end up with several next-hops for the same destination, and then it has to figure out which next-hop to use for each data packet. that's the whole main character problem 👑

1. Introduction ✨

so OSPF and ISIS both explicitly allow "Equal-Cost Multipath" routing, and some routers do it with RIP too 💜 the point is that if there are multiple equal-cost routes to the same destination, they can all be found and used to load balance across redundant paths. iconic energy 🦋

the catch 🙈 is that a forwarder with multipath might have several next-hops for any destination, and it has to use some method to pick which one each packet goes to. this memo is basically a recap of what people do right now, what goes wrong, and what the fixes are 🫶

2. Concerns 💀

some routers allow multipath forwarding, and a lot of them do it naively with round-robin 🙃 that means each packet matching a given destination route goes out on the next next-hop in line. it does give you a form of load balancing, but round-robin (and random too) has some real problems 😭

Variable Path MTU 🎀 since each redundant path might have a different MTU, the overall path MTU can change packet by packet 😱 which kind of ruins the whole point of path MTU discovery. not it. 💅

Variable Latencies 🌸 each redundant path might have different latency, so letting packets take separate paths can make them arrive out of order 🌀 that means more delivery latency and more buffering needed.

the reordering part is spicy 🌶️ TCP thinks loss happened when packets with higher sequence numbers show up before an earlier one. when three or more packets arrive before a "late" packet, TCP goes into a mode called "fast-retransmit" [6] 📉 that burns extra bandwidth (which could cause more loss and lower throughput) because it tries to retransmit the delayed packet(s) for no reason. so yes, reordering can be detrimental to network performance 😤

Debugging 🔍 common debugging tools like ping and traceroute get way less reliable with multiple paths, and they can even show completely wrong results 🙈

now for multicast 🧚 the issue there is that multicast routing protocols prevent loops and duplicates by building a single tree to all receivers of the same group address. the multicast protocols deployed today (DVMRP, PIM-DM, PIM-SM) [2] build shortest-path trees rooted either at the source or at another router called a Core or Rendezvous Point 🌳 so the way they avoid duplicates is that a given tree has to use only one next-hop toward the root of the tree. iykyk 💜

3. Requirements 🎀

okay so in the rest of this memo we use the word "flow" 💖 it means the granularity at which a router keeps state (if it keeps any) for classes of traffic. the exact definition can depend on the implementation. a flow might be identified just by destination address, or by a (source address, destination address, protocol id) triplet 🦋

"flow" is not the same thing as the "microflow" from RFC 2474 [7], which also includes port numbers ✨ and including transport-layer info in next-hop selection can actually be problematic 😬 for example, if packets get fragmented, the transport-layer info might not be in every packet. and choosing a path based on transport-layer fields could ruin the benefit of caching things like MTU for later connections between the same endpoints 💔

every problem from the last section happens when packets in the same unicast or multicast "flow" get split across multiple paths. so the natural fix is to make sure packets for the same flow always take the same path 👑

there are also two extra features we want 🫶

Minimal disruption 🌷 when multipath is in use, meaning multiple routes contribute valid next-hops, the chances go up that routes get added and deleted from consideration. compare that with only using the "best" route, where metric changes on alternate routes don't affect traffic paths at all 🙅‍♀️ since more routes may actually be used for forwarding under multipath, the potential for packet reordering and loss from route flaps can be way bigger than without multipath. so it's desirable to minimize how many active flows get affected when another next-hop is added or deleted 💅

Fast implementation ⚡ the extra computation needed to forward a packet should be small 🧚 for example, with round-robin this computation might just be incrementing a next-hop index (modulo the number of next-hops) 🎀

4. Solutions 🌸

we're going to give three possible methods for improving multipath performance, and then talk about how they apply to unicast and multicast forwarding 🦋

Modulo-N Hash 🔥 to pick a next-hop from a list of N next-hops, the router does a modulo-N hash over the packet header fields that identify a flow. the big upside is that it's fast 🏎️ the downside is that whenever a next-hop is added or removed, (N-1)/N of all flows change paths 😭 that's (N-1)/N, bestie, so a lot.

Hash-Threshold 💜 the router first picks a key by hashing the packet header fields that identify the flow. the N next-hops each get their own unique region in the hash function's output space ✨ then the router compares the hash value against the region boundaries to see which region it falls into, and that tells it which next-hop to use. the advantage is that when next-hops are added or removed, only flows near the region boundaries (or thresholds) get affected 🙌 for ECMP, the lookup can be a simple division: hash_value / fixed_region_size. when a next-hop is added or removed, between 1/4 and 1/2 of all flows change paths. an analysis of this method is in [3] 📚

Highest Random Weight (HRW) 👑 the router computes a key for EACH next-hop by hashing the packet header fields that identify the flow plus the address of the next-hop 🧚 then it picks the next-hop with the highest key value [4]. this is the main character move because it minimizes how many flows are affected by a next-hop addition or deletion (only 1/N of them!) 😍 the tradeoff is that it's approximately N times as expensive as a modulo-N hash 🫠

the applicability of these three options depends on (at least) two factors: whether the forwarder keeps per-flow state, and how precious CPU is to a multipath forwarder 💅

some routers keep per-flow state for reasons unrelated to multipath 🌈 for example, routers typically keep per-flow state for multicast flows so they can track which interfaces packets in the flow should be copied to 🦋

if a multipath forwarder does keep per-flow state, the router can compute the next-hop at state creation time ✨ that means no extra computation at packet forwarding time compared with normal forwarding to a single next-hop, since the next-hop is precomputed 🙌 in that case, any method can be used: round-robin, random, modulo-N, hash-threshold, or HRW 🎀

hash functions like modulo-N, hash-threshold, and HRW are better if the forwarder state might get deleted for any reason during a flow's lifetime 🧠 the reason is that later next-hop computations by the router will always select the same path 💖 this also makes debugging tools like traceroute more useful 🔍 and to maximize path stability (and so the usefulness of traceroute, etc.), the use of HRW is recommended over the other methods mentioned herein 👑

if the forwarder doesn't keep per-flow state, then using multiple next-hops means the next-hop has to be calculated when the packet arrives 📦 when CPU is more precious than stability of flow paths, hash-threshold is recommended over the other methods mentioned herein 💅

4.1. Unicast Forwarding 🌷

depending on the implementation, unicast forwarding may or may not keep per-flow state 🦋 our recommendation is this: where forwarder implementations keep flow state, routers should use HRW at state creation time (and next-hop deletion time) to select the next-hop, and forwarders without per-flow state should use hash-threshold 💖

4.2. Multicast Forwarding 🧚

today's multicast forwarding engines use a cache of forwarding entries indexed by group (or group prefix) and source (or source prefix) ✨ which means they always keep per-flow state 🙌 for some multicast routing protocols, the "flow" can be pretty coarse, like traffic from all sources going to the same destination 🌊 since the forwarder keeps per-flow state, the recommendation is that the router always use HRW to select the next-hop 👑

routers using explicit-joining protocols such as PIM-SM [5] should use the multipath information when deciding which neighbor a join message gets sent to 📮 for example, when there are multiple next-hops for a given Rendezvous Point (RP) that a (*,G) Join should go toward, it is recommended that HRW be used to pick the next-hop for each group 🫶

okay so this one is about what happens when two routers are linked by multiple parallel links 🔗 bestie, the usual move is to bundle those links together into one "super"-link and then route over that. ngl it's very clean 💅 For multicast forwarding, the two links end up looking like a single next-hop over the combined link, and that's what lets you stop duplicates from showing up 🙈

Here's the catch: when a unicast or multicast packet gets queued onto the combined link, you still need some method, like the ones we already covered, to pick the physical link the packet actually goes out on 🎀 Lowkey the big stuff from this document mostly goes away IF the parallel links are identical 🫶 The one exception is packet reordering, which can still happen with round-robin and that's not great for TCP 😭

7. Security Considerations 🔐

okay this section is giving very important vibes 💀 This document is about the different ways to choose one next-hop out of several valid ones. So it doesn't directly affect the security of the Internet infrastructure or the apps running on it 🌸

BUT there's one thing worth mentioning! If next-hop selection is predictable, an attacker can craft traffic that all hashes to the same place 🙈 That means they could overload one particular path and cause a denial-of-service attack. The worst version is when that same single next-hop gets picked every time, so this kind of attack is easiest when multipath isn't being used at all 😬 Introducing multipath routing can make it harder to pull off, because the more unpredictable the hash is, the harder it gets to DoS any single link 🦋

8. References 📚

  • [1] Moy, J., "OSPF Version 2", STD 54, RFC 2328, April 1998.
  • [2] Maufer, T., "Deploying IP Multicast in the Enterprise", Prentice-Hall, 1998.
  • [3] Hopps, C., "Analysis of an Equal-Cost Multi-Path Algorithm", RFC 2992, November 2000.
  • [4] Thaler, D., and C.V. Ravishankar, "Using Name-Based Mappings to Increase Hit Rates", IEEE/ACM Transactions on Networking, February 1998.
  • [5] Estrin, D., Farinacci, D., Helmy, A., Thaler, D., Deering, S., Handley, M., Jacobson, V., Liu, C., Sharma, P. and L. Wei, "Protocol Independent Multicast-Sparse Mode (PIM-SM): Protocol Specification", RFC 2362, June 1998.
  • [6] Allman, M., Paxson, V. and W. Stevens, "TCP Congestion Control", RFC 2581, April 1999.
  • [7] Nichols, K., Blake, S., Baker, F. and D. Black., "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998.

9. Authors' Addresses 🧚

Dave Thaler
Microsoft
One Microsoft Way
Redmond, WA  98052

Phone: +1 425 703 8835
EMail: dthaler@dthaler.microsoft.com


Christian E. Hopps
NextHop Technologies, Inc.
517 W. William Street
Ann Arbor, MI 48103-4943
U.S.A

Phone: +1 734 936 0291
EMail: chopps@nexthop.com

Copyright (C) The Internet Society (2000). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 🌈

Funding for the RFC Editor function is currently provided by the Internet Society.

📦 where this page comes from