RFC 2439
BGP Route Flap Damping
Network Working Group C. Villamizar
Request for Comments: 2439 ANSBGP Route Flap Damping ✨🦋
Hiii bestie 🫶 okay so this is RFC 2439, and it's giving "BGP Route Flap Damping." It was written by C. Villamizar (ANS), R. Chandra (Cisco), and R. Govindan (ISI), and dropped in November 1998. It's a Standards Track document, which means it's an actual internet standards-track protocol spec for the whole internet community, not just vibes. It asks for discussion and suggestions for improvements, and you check STD 1 ("Internet Official Protocol Standards") for the official standardization state and status. Distribution is unlimited, and The Internet Society owns the copyright (1998), all rights reserved 💅.
Nothing is listed as obsoleting or updating anything in this title block, so this one is a fresh start 🌸.
Status of This Memo ✨
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice 💜
Copyright (C) The Internet Society (1998). All Rights Reserved.
Abstract 🦋
Okay so the big idea here is a way to use BGP that cuts down on the routing traffic you pass along to your routing peers 📉. That means less load on those peers too, and the cool part is it doesn't wreck how fast stable routes converge ⚡. This technique is already baked into commercial BGP products 🔥, and it also works for IDRP 🧚.
The overall goals are:
- to provide a mechanism capable of reducing router processing load caused by instability
- in doing so prevent sustained routing oscillations
- to do so without sacrificing route convergence time for generally well behaved routes
And this has to happen while keeping the other BGP goals in mind, obvi 🙈:
- pack changes into a small number of updates
- preserve consistent routing
- minimal addition space and computational overhead
So like, there's been an excessive rate of updates to the advertised reachability of a subset of Internet prefixes everywhere 😭. People in Internet operations noticed this in the early 1990s, and it's still happening. These updates aren't necessarily periodic, so "route oscillation" is kind of a misleading name 🙃. The informal term everyone uses is "route flap", and the techniques in this doc are now widely deployed and commonly called "route flap damping" 🎀.
1. Overview 🌈
Okay so to keep a routed internet scalable, you have to cut down how much routing-state change BGP passes along, so processing doesn't explode 💥. The main sources of processing load from BGP updates are the BGP decision process and adding and removing forwarding entries.
Picture this: a widely deployed BGP implementation can just... fail when the update volume gets too high 😵. For example, it might not be able to keep its BGP or IGP sessions alive if it's overloaded. One router failing can pile more load onto the others 📚. That extra load can knock out other copies of the same implementation, or other implementations with a similar weakness 💀. In the worst case you get a stable oscillation, and those worst cases have already happened in the real world 😭.
A BGP implementation has to be ready for a huge volume of routing traffic 🏋️♀️. It can't count on the sender to shield it from route instability. The guidelines here are meant to stop sustained oscillations, but they don't replace the need for robust and efficient implementations 💅. The mechanisms described here let routing instability get contained at the AS border router that borders the instability 🛡️.
Even when BGP implementations are super robust, the routing process still has performance limits 📉. So limiting unnecessary change becomes about keeping route change convergence time reasonable as the routing topology grows 🌱.
2. Methods of Limiting Route Advertisement 🎀
Two methods for controlling how often routes get advertised are described here ⏱️. The first uses fixed timers. Fixed timers have no space overhead per route, but the downside is they slow convergence for the normal case, where a route has no history of instability 🐢. The second method fixes that limitation, but it costs some extra space. That overhead is a small amount of state per route plus a very small processing overhead 🧠.
It's possible, and desirable, to combine both techniques 🫶. In practice, fixed timers have been set to very short intervals, and they've proven useful for packing routes into fewer updates when routes arrive in separate updates 📦. BGP calls this packing Network Layer Reachability Information (NLRI) [5] ✨.
Fixed timers are rarely set to the tens of minutes or hours you'd need to actually damp route flap 🙅♀️. Doing that would severely limit routing convergence, which is the undesirable part 😤.
2.1. Existing Fixed Timer Recommendations 💖
BGP-3 doesn't make specific recommendations in this area [1] 🤷♀️. The short section called "Frequency of Route Selection" just recommends that something be done, and it makes broad statements about certain properties that are desirable or undesirable 🙄.
BGP4 keeps the "Frequency of Route Advertisement" section and adds a "Frequency of Route Origination" section 🆕. BGP-4 describes a way to limit route advertisement using a fixed (configurable) MinRouteAdvertisementInterval timer and a fixed MinASOriginationInterval timer [5] ⏲️. The recommended timer values are 30 seconds for MinRouteAdvertisementInterval and 15 seconds for MinASOriginationInterval 🎀.
2.2. Desirable Properties of Damping Algorithms 🦋
Before we get into damping algorithms, we need to be super clear on the objectives 🎯. Some key properties get examined so the design rationale makes sense 📝.
The overall objective is to reduce the route update load without limiting convergence time for well behaved routes ✨. To do that, you need criteria for well behaved and poorly behaved routes 🧐. You also need an algorithm that can identify the poorly behaved ones 🔍. Ideally, that measure would predict the route's future stability 🔮.
Any delay in propagating well behaved routes should be minimal 🏃♀️. A bit of delay is tolerable, though, because it helps pack updates better 📦. Delay for poorly behaved routes should, if possible, be proportional to a measure of how unstable the route is expected to be in the future 📈. Delaying an unstable route should suppress it until there's some confidence that it has stabilized 🫡.
If a bunch of route changes come in as separate updates over a very short period, and they could be combined into one update, then pack them as efficiently as possible before propagating further 💅. A small delay for well behaved routes is tolerable, and it's needed so updates can be packed better 🫶.
When routes are unstable, you should suppress their use and announcement, not their removal 🙅♀️. If one route to a destination is stable and another is somewhat unstable, the unstable one should, if possible, be suppressed more aggressively than it would be if there were no alternate path 👑.
Routing consistency inside an AS is very important, so only very minimal delay of internal BGP (IBGP) should happen 🏠. Routing consistency across AS boundaries matters just as much 🌍. It's highly undesirable to advertise a route that differs from the route actually in use, except for a very minimal time 🙈. It's more desirable to suppress accepting a route (and therefore using it in the IGP) than to suppress only the redistribution 🔥.
It's clearly not possible to predict a route's future stability accurately 🔮🙃. Recent stability history is generally regarded as a good basis for estimating future stability 📚. So the criteria for telling well behaved from poorly behaved routes is based on that recent history 🕰️. There's no simple quantitative expression of recent stability, so a figure of merit has to be defined 🧮. A good figure of merit would have these traits: the further in the past an instability happened, the less it affects the figure of merit, and the instability measure is cumulative rather than only reflecting the latest event ➕.
The algorithms should behave so that routes with a history of stability that make a few transitions get those transitions made quickly ⚡. If transitions keep coming, advertisement of the route should be suppressed 🚫. There should be some memory of prior instability 🧠. How much prior instability counts should gradually shrink as long as the route stays announced and stable 📉.
2.3. Design Choices 🎀
After routes have been accepted, their readvertisement is briefly suppressed to improve packing of updates 📦. There may be a lengthy suppression of accepting an external route 🕐. How long a route gets suppressed is based on a figure of merit that's expected to correlate with the probability of future instability 📊. Routes with high figure of merit values get suppressed 🙅♀️. An exponential decay algorithm was chosen as the basis for reducing the figure of merit over time 📉. These choices should be treated as suggestions for implementation 💡.
An exponential decay function has the property that previous instability can be remembered for a fairly long time 🧠. The rate at which the instability figure of merit decays slows down as time goes on 🐌. Exponential decay has this property:
f(f(figure-of-merit, t1), t2) = f(figure-of-merit, t1+t2)This property means you can compute the decay over a long period in a single operation, no matter what the current figure-of-merit value is 🧮✨. As a performance optimization, the decay can be applied in fixed time increments 🏎️. Given a desired decay half life, the decay for a single time increment can be computed ahead of time 🗓️. The decay for multiple time increments is expressed below:
f(figure-of-merit, n*t0) = f(figure-of-merit, t0)**n = K**nYou can precompute the values of K ** n for a reasonable number of "n" and store them in an array 📚. The value of "K" is always less than one 🔢. The array size can be bounded because the value quickly approaches zero 🎯. That makes the decay easy to compute with an array bound check, an array lookup, and a single multiply, no matter how much time has passed 💖🦋.
3. Limiting Route Advertisements using Fixed Timers ⏰
okay so this one is all about fixed timers on the process of sending routes 🎀 Its main job is to pack routes nicely into BGP update messages. A stable route's delay before it gets advertised should be bounded and minimal ✨ For an unreachable route the delay doesn't have to be zero, but it should also be bounded, and it should probably have its own bound that's less than or equal to the one for a reachable advertisement 💅
BGP defines a Routing Information Base (RIB) 🧚 Routes that need to be readvertised can get marked right in the RIB, or you can keep a separate set of structures that points back into the RIB. Either way works 🫶
Every so often, a subset of the marked routes gets flushed. It's fairly straightforward and it gets the job done 🌸 But if you make the implementation too simple, the computation can end up being order N squared 😭 To avoid that N squared performance, you need some kind of data structure that groups routes with common attributes. Main character energy, but for data structures 👑
An implementation should pack updates efficiently and provide a minimum readvertisement delay. It should also provide a bound on the maximum readvertisement delay that would be experienced solely as a result of the algorithm used to provide a minimum delay, and it must be computationally efficient even when there's a very large number of candidates for readvertisement 🔥
4. Stability Sensitive Suppression of Route Advertisement 🦋
this method is all about a measure of route stability, applied on a per route basis 💜 It only gets applied when receiving updates from external peers (EBGP) ✨ Applying it to IBGP learned routes, or to advertisements sent to IBGP or EBGP peers after a route selection, can cause routing loops 💀 so please don't 🙈
Each route keeps a figure of merit based on instability 🍓 That figure of merit is what decides whether the route gets suppressed. Routes with a high figure of merit get suppressed. Every time a route is withdrawn, its figure of merit goes up ⬆️ While the route isn't changing, the figure of merit decays exponentially, and the decay rate depends on whether the route is stable and reachable, or has been stable and unreachable 🌈 The decay rate can be slower when the route is unreachable, or the figure of merit can just stay fixed (no decay at all) while the route stays unreachable. Whether unreachable routes decay at the same rate, a slower rate, or not at all is an implementation choice. Decaying at a slower rate is recommended 🫧
A very efficient implementation is suggested in the sections that follow 🧠 It only does computation for the routes contained in an update, at the moment an update is received or withdrawn. That's a big difference from the simplistic approach of periodically decaying every single route. The suggested implementation is only a small number of simple operations, and it can be done with scaled integers ✨
The behavior of unstable routes is pretty predictable 🔮 Severely flapping routes often get advertised and withdrawn at regular time intervals that match the timers of a particular protocol (the IGP or exterior protocol in use where the problem exists). Marginal circuits or mild congestion can cause a long term pattern of occasional brief route withdrawals or occasional brief connectivity 🥹
4.1. Single vs. Multiple Configuration Parameter Sets 🎀
the algorithm's behavior gets shaped by a bunch of configurable parameters 🎛️ You can set up separate sets of parameters for two different problems: short term severe route flap, and chronic milder route flap (a pattern of occasional drops over a long time period) 💅 The first one needs a fast decay and a low threshold. That lets a small number of consecutive flaps suppress a route, but it can be reused again after a relatively short stretch of stability. The second one needs a very slow decay and a higher threshold, and it might make sense for routes that have an alternate path of similar bandwidth 🌸
It can also be desirable to set different thresholds for routes with roughly equivalent alternate paths than for routes where the alternate paths have lower bandwidth or tend to be congested 🫶 You can handle this by associating a different set of parameters with different ranges of preference values. Parameter selection could be based on BGP LOCAL_PREF 💜
Parameter selection could also be based on whether an alternate route was known 👀 A route would be considered if, for any applicable parameter set, an alternate route with the specified preference value existed and the figure of merit for that parameter set did not indicate a need to suppress the route. A less aggressive suppression would apply when no alternate route existed at all. In the simplest case, a more aggressive suppression would apply if any alternate route existed. Only the highest preference (most preferred) value needs to be specified, since the ranges may overlap 🧚
It might also be desirable to configure a different set of thresholds for routes that rely on switched services and may disconnect at times to reduce connect charges 📞 Those routes might be expected to change state somewhat more often, but they should still be suppressed if continuous state changes indicate instability ⚠️
While it's not essential, it might be desirable to configure multiple sets of configuration parameters per route 🎀 It may also be desirable to configure sets of parameters that only correspond to a set of routes, identified by AS path, peer router, specific destinations, or other means 🦋 Experience may dictate how much flexibility is needed and how best to set the parameters. Whether to allow different damping parameter sets for different routes, and whether to allow multiple figures of merit per route, is an implementation choice 🫧
Parameter selection can also be based on prefix length 📏 The reasoning is that longer prefixes tend to reach fewer end systems and are less important, so these less important prefixes can be damped more aggressively. This technique is in fairly widespread use 🌈 Small sites, or sites with dense address allocation that are multihomed, are often reachable by long prefixes that aren't easily aggregated. These sites tend to dispute the choice of prefix length for parameter selection 😤 Advocates of the technique point out that it encourages better aggregation 💖
4.2. Configuration Parameters ⚙️
at configuration time, the user can specify a number of parameters 🧁 Configuration parameters are expressed in units that are meaningful to the user. These differ from the parameters used at run time, which are in units that are convenient for computation. The run time parameters are derived from the configuration parameters. The suggested configuration parameters are listed below 📝
cutoff threshold (cut)- This value is expressed as a number of route withdrawals. It is the value above which a route advertisement will be suppressed.
reuse threshold (reuse)- This value is expressed as a number of route withdrawals. It is the value below which a suppressed route will now be used again.
maximum hold down time (T-hold)- This value is the maximum time a route can be suppressed, no matter how unstable it has been prior to this period of stability.
decay half life while reachable (decay-ok)- This value is the time duration, in minutes or seconds, during which the accumulated stability figure of merit will be reduced by half if the route is considered reachable (whether suppressed or not).
decay half life while unreachable (decay-ng)- This value is the time duration, in minutes or seconds, during which the accumulated stability figure of merit will be reduced by half if the route is considered unreachable. If not specified or set to zero, no decay will occur while a route remains unreachable.
decay memory limit (Tmax-ok or Tmax-ng)- This is the maximum time that any memory of previous instability will be retained, given that the route's state remains unchanged, whether reachable or unreachable. This parameter is generally used to determine array sizes.
There may be multiple sets of the parameters above, as described in Section 4.1. The configuration parameters listed below would be applied system wide. These include the time granularity of all computations, and the parameters used to control reevaluation of routes that have previously been suppressed 🌟
time granularity (delta-t)- This is the time granularity in seconds used to perform all decay computations.
reuse list time granularity (delta-reuse)- This is the time interval between evaluations of the reuse lists. Each reuse list corresponds to an additional time increment.
reuse list memory reuse-list-max- This is the time value corresponding to the last reuse list. This may be the maximum value of T-hold for all parameter sets, or it may be configured.
number of reuse lists (reuse-list-size)- This is the number of reuse lists. It may be determined from reuse-list-max or set explicitly.
A recommended optimization is described in Section 4.8.6 🪄 It involves an array referred to as the "reuse index array". A reuse index array is needed for each decay rate in use. The reuse index array is used to estimate which reuse list to place a route in when it is suppressed. Proper placement avoids the need to periodically evaluate decay to figure out whether a route can be reused or when storage can be recovered 🙌 Using the reuse index array also avoids the need to compute a logarithm to determine placement. One additional system wide parameter can be introduced:
reuse index array size (reuse-index-array-size)- This is the size of reuse index arrays. This size determines the accuracy with which suppressed routes can be placed within the set of reuse lists when they are suppressed for a long time.
4.3. Guidelines for Setting Parameters 📐
the decay half life should be set to a time considerably longer than the period of the route flap it's meant to address ⏳ For example, if the decay is set to ten minutes and a route is withdrawn and readvertised exactly every ten minutes, the route would keep flapping if the cutoff was set to a value of 2 or above 😭
The stability figure of merit is itself an accumulated, time decayed total 🧮 This has to be kept in mind when setting the decay time, cutoff values, and reuse values. The figure of merit goes up each time a route transitions from reachable to unreachable. It's decayed at a rate proportional to its current value. So increasing the rate of route flap increments the figure of merit more often, and it reaches a given threshold in a shorter amount of time 🚀 When you plot the response to a constant rate route flap, it looks like a sawtooth with an abrupt rising edge and a decaying falling edge 🦷 Since the absolute decay amount is proportional to the figure of merit, at a continuous constant flap rate the baseline of the sawtooth will tend to stop rising and converge, unless it's clipped by a ceiling value 🫶
If the ceiling clips the sawtooth, the baseline just hits the ceiling sooner when the route keeps flapping. Here's an example at four times the decay rate 🎢. The first time the route goes unreachable, the value becomes 1. Each later flap adds one to the previous value, which has already decayed by the fourth root of 2 (the amount that would decay in 1/4 of the half-life if decay is exponential). The sequence is 1, 1.84, 2.55, 3.14, 3.64, 4.06, 4.42, 4.71, 4.96, 5.17, ..., converging at about 6.285. At that flap rate the route hits 3 after 4 cycles, 4 after 6 cycles, 5 after 10 cycles, and settles around 6.3 💅. At twice the decay time, it hits 3 after 7 cycles and converges below 3.5 ✨.
Figure 1 shows the stability figure of merit for a route flapping at a constant rate 📈. The time axis is in multiples of the decay half-life, and the plots show flap periods of 1/2, 1/3, 1/4, and 1/8 of the half-life. A ceiling of 4.5 was set, which clips three of the plots and limits how long it takes to re-advertise the route no matter what its history was 🙈. With cutoff and reuse thresholds of 1.5 and 0.75, routes get suppressed after being declared unreachable 2-3 times, and they come back after about 2 decay half-life periods of stability 🦋.
This function can be written out formally 🤓. Reachability is a variable R that's either 0 (unreachable) or 1 (reachable), and at any discrete time it can only have one value. The figure of merit goes up by 1 on every transition from R=1 to R=0, and it gets clipped at the ceiling. The decay over discrete times looks like this:
figure-of-merit(t) = K * figure-of-merit(t - delta-t)
K = K1 for R=0 K=K2 for R=1The four plots are stacked vertically. There's not enough room for every point on the time axis, so only some are shown, along with the figure of merit value. Next to each value is a super low-res strip chart made of ASCII dots 🌸. It's only there to give you a vibe for the rise and fall. The strip charts aren't drawn on overlapping axes because the sawtooth waves cross each other a lot, and at this resolution the rise and fall of the baseline shows up but the sawtooth shape only shows in the printed numbers 👀.
From the maximum hold time (T-hold), you can work out the ratio of the reuse value to the ceiling 🧮. You can then pick an integer ceiling big enough that overflow won't be an issue, and scale everything else to match. If both cutoffs are specified, or if multiple parameter sets are used, the highest ceiling is the one that gets used 👑.
time figure-of-merit as a function of time (in minutes)
0.00 0.000 . 0.000 . 0.000 . 0.000 .
0.08 0.000 . 0.000 . 0.000 . 0.000 .
0.16 0.000 . 0.000 . 0.000 . 0.973 .
0.24 0.000 . 0.000 . 0.000 . 0.920 .
0.32 0.000 . 0.000 . 0.946 . 1.817 .
0.40 0.000 . 0.953 . 0.895 . 2.698 .
0.48 0.000 . 0.901 . 0.847 . 2.552 .
0.56 0.953 . 0.853 . 1.754 . 3.367 .
0.64 0.901 . 0.807 . 1.659 . 4.172 .
0.72 0.853 . 1.722 . 1.570 . 3.947 .
0.80 0.807 . 1.629 . 2.444 . 4.317 .
0.88 0.763 . 1.542 . 2.312 . 4.469 .
0.96 0.722 . 1.458 . 2.188 . 4.228 .
1.04 1.649 . 2.346 . 3.036 . 4.347 .
1.12 1.560 . 2.219 . 2.872 . 4.112 .
1.20 1.476 . 2.099 . 2.717 . 4.257 .
1.28 1.396 . 1.986 . 3.543 . 4.377 .
1.36 1.321 . 2.858 . 3.352 . 4.141 .
1.44 1.250 . 2.704 . 3.171 . 4.287 .
1.52 2.162 . 2.558 . 3.979 . 4.407 .
1.60 2.045 . 2.420 . 3.765 . 4.170 .
1.68 1.935 . 3.276 . 3.562 . 4.317 .
1.76 1.830 . 3.099 . 4.356 . 4.438 .
1.84 1.732 . 2.932 . 4.121 . 4.199 .
1.92 1.638 . 2.774 . 3.899 . 3.972 .
2.00 1.550 . 2.624 . 3.688 . 3.758 .
2.08 1.466 . 2.483 . 3.489 . 3.555 .
2.16 1.387 . 2.349 . 3.301 . 3.363 .
2.24 1.312 . 2.222 . 3.123 . 3.182 .
2.32 1.242 . 2.102 . 2.955 . 3.010 .
2.40 1.175 . 1.989 . 2.795 . 2.848 .
2.48 1.111 . 1.882 . 2.644 . 2.694 .
2.56 1.051 . 1.780 . 2.502 . 2.549 .
2.64 0.995 . 1.684 . 2.367 . 2.411 .
2.72 0.941 . 1.593 . 2.239 . 2.281 .
2.80 0.890 . 1.507 . 2.118 . 2.158 .
2.88 0.842 . 1.426 . 2.004 . 2.042 .
2.96 0.797 . 1.349 . 1.896 . 1.932 .
3.04 0.754 . 1.276 . 1.794 . 1.828 .
3.12 0.713 . 1.207 . 1.697 . 1.729 .
3.20 0.675 . 1.142 . 1.605 . 1.636 .
3.28 0.638 . 1.081 . 1.519 . 1.547 .
3.36 0.604 . 1.022 . 1.437 . 1.464 .
3.44 0.571 . 0.967 . 1.359 . 1.385 .Figure 1: Instability figure of merit for flap at a constant rate
time figure-of-merit as a function of time (in minutes)
0.00 0.000 . 0.000 . 0.000 .
0.20 0.000 . 0.000 . 0.000 .
0.40 0.000 . 0.000 . 0.000 .
0.60 0.000 . 0.000 . 0.000 .
0.80 0.000 . 0.000 . 0.000 .
1.00 0.999 . 0.999 . 0.999 .
1.20 0.971 . 0.971 . 0.929 .
1.40 0.945 . 0.945 . 0.809 .
1.60 0.919 . 0.865 . 0.704 .
1.80 0.894 . 0.753 . 0.613 .
2.00 1.812 . 1.657 . 1.535 .
2.20 1.762 . 1.612 . 1.428 .
2.40 1.714 . 1.568 . 1.244 .
2.60 1.667 . 1.443 . 1.083 .
2.80 1.622 . 1.256 . 0.942 .
3.00 1.468 . 1.094 . 0.820 .
3.20 2.400 . 2.036 . 1.694 .
3.40 2.335 . 1.981 . 1.475 .
3.60 2.271 . 1.823 . 1.284 .
3.80 2.209 . 1.587 . 1.118 .
4.00 1.999 . 1.381 . 0.973 .
4.20 2.625 . 2.084 . 1.727 .
4.40 2.285 . 1.815 . 1.503 .
4.60 1.990 . 1.580 . 1.309 .
4.80 1.732 . 1.375 . 1.139 .
5.00 1.508 . 1.197 . 0.992 .
5.20 1.313 . 1.042 . 0.864 .
5.40 1.143 . 0.907 . 0.752 .
5.60 0.995 . 0.790 . 0.654 .
5.80 0.866 . 0.688 . 0.570 .
6.00 0.754 . 0.599 . 0.496 .
6.20 0.656 . 0.521 . 0.432 .
6.40 0.571 . 0.454 . 0.376 .
6.60 0.497 . 0.395 . 0.327 .
6.80 0.433 . 0.344 . 0.285 .
7.00 0.377 . 0.299 . 0.248 .
7.20 0.328 . 0.261 . 0.216 .
7.40 0.286 . 0.227 . 0.188 .
7.60 0.249 . 0.197 . 0.164 .
7.80 0.216 . 0.172 . 0.142 .
8.00 0.188 . 0.150 . 0.124 .Figure 2: Separate decay constants when unreachable
Figure 2 shows what happens when you set separate decay rates for when the route is reachable and when it's unreachable 🎀. Here the decay is 5 times slower while the route is unreachable. The three cases all have a flap period equal to the decay half-life, but the route is reachable 1/8 of the time in one case, 1/2 the time in another, and 7/8 of the time in the last 😭. In that last case the route doesn't get suppressed until after the third unreachable event (when it's above the top threshold after coming back).
The main point of Figure 2 is the effect of changing the duty cycle of the square wave in R while the frequency stays fixed 🌈. If decay is slower when R=0 (the route is unreachable), the figure of merit climbs more slowly as the route spends more time reachable. More precisely, the baseline of the sawtooth climbs more slowly. When the route becomes steadily reachable again, the difference can be pretty negligible if the sawtooth is clipped at the ceiling. It matters more when the flapping is slow or short, so the sawtooth never reaches the ceiling. In Figure 2 the unstable window is short enough that the ceiling never gets hit, so routes that are reachable for a bigger share of the flap cycle get reused (put in the RIB and advertised to peers) sooner than the others once they're stable again. That's when R becomes 1 and the announced state stays reachable 🧚.
In both Figure 1 and Figure 2, routes get suppressed. Routes flapping at the decay half-life or faster get withdrawn two or three times, then stay withdrawn until they've been stably announced for roughly 1 1/2 to 2 1/2 times the decay half-life (with the ceiling from the example) 💜.
The whole reason for damping BGP route flap is to cut the processor load on the router itself and on everything downstream, meaning BGP peers and peers of peers that see the announcements this router sends out 🫶. Computing a figure of merit at every discrete time step with figure-of-merit(t) = K * figure-of-merit(t - delta-t) would be super inefficient and would defeat the purpose 💀. The fix is to put off computation as long as possible, then do one simple calculation that accounts for all the decay since the figure of merit was last updated. Decay arrays give you that single simple calculation. Reuse lists (covered later) are how you postpone the calculations. A route becomes usable if nothing changes for a while and it's unreachable. The data structure storage gets recovered if the route's state hasn't changed for a while and it's been unreachable. The reuse arrays help estimate how long a computation can be postponed if nothing else changes 🍓✨.
Okay so bestie, a bigger time granularity keeps the table storage down 🧚 The granularity should be less than the shortest reasonable gap between the worst-case route flaps you'd expect 💅 It might be reasonable to fix this parameter at compile time, or set a default and strongly recommend that the user leave it alone 🫶
With exponential decay, the array can get way smaller if you set a period of total stability after which the decayed total counts as zero instead of hanging onto a tiny leftover number 🦋 Alternatively, super long decays can be done by multiplying more than once if you hit the array bounds 🌸
Now the reuse lists! 👑 These hold suppressed routes grouped by how long it'll be until they're eligible for reuse. Periodically, each list moves forward one position and one list gets removed, as described in Section 4.8.7 ✨ All the suppressed routes in the removed list get reevaluated and either used, or dropped into another list depending on how much more time has to pass before the route can be reused. The last list always holds every route that won't be advertised for more time than fits in the remaining list heads. When the last list reaches the front, some routes won't be ready yet and will have to be requeued 😭 The time interval for reconsidering suppressed routes and the number of list heads should be configurable. Reasonable defaults might be 30 seconds and 64 list heads. A route suppressed for a long time would need to be reevaluated every 32 minutes 🙈
4.4 Run Time Data Structures
Here's the storage situation 💜 A fixed, small amount of per-system storage will be required. Where sets of multiple configuration parameters are used, storage will be required per set of parameters. A small amount of per-route storage is required too. You also need a set of list heads, which are used to arrange suppressed routes by the time remaining until they can be reused 🎀
A separate reuse list can be used to hold unreachable routes, so you can recover their storage if they stay unreachable too long. Honestly this might be more accurately called a recycling list ♻️ The advantage is that free data structures become available as soon as possible. Or, you can just put the data structures on a queue and recover the storage when the route reaches the front of the queue and storage is actually needed. The latter is less optimal but way simpler ✨
If multiple sets of configuration parameters are allowed per route, you need some way to associate more than one figure of merit and set of parameters with each route. Building a linked list of these objects is one of several reasonable implementations 🌈 Similarly, you need a way to associate a route with a reuse list. A small overhead comes with the pointers needed for whatever data structure is chosen for the reuse lists. The suggested implementation uses double linked lists, so it requires two pointers per figure of merit 🍓
Each set of configuration parameters can reference decay arrays and reuse arrays. These arrays should be shared among multiple sets of parameters, since their storage requirement is not negligible 😳 There will be only one set of reuse list heads for the entire router 👑
4.4.1 Data Structures for Configuration Parameter Sets
Based on the configuration parameters described in the previous section, the following values can be computed as scaled integers directly from the corresponding configuration parameters 🧮
- decay array scale factor (
decay-array-scale-factor) - cutoff value (
cut) - reuse value (
reuse) - figure of merit ceiling (
ceiling)
Each configuration parameter set will reference one or two decay arrays and one or two reuse arrays. Only one array will be needed if the decay rate is the same while a route is unreachable as while it's reachable, or if the stability figure of merit doesn't decay while a route is unreachable 💅
4.4.2 Data Structures per Decay Array and Reuse Index Array
These ones are also computed from the configuration parameters, just not as directly. The computation is described in Section 4.5 🦋
- decay rate per tick (
decay-delta-t) - decay array size (
decay-array-size) - decay array (
decay[]) - reuse index array size (
reuse-index-array-size) - reuse index array (
reuse-index-array[])
For each decay rate specified, an array stores the value of a computed parameter raised to the power of each array element's index. This is to speed up computations 🚀 The decay rate per tick is an intermediate value, expressed as a real number, that's used to compute the values stored in the decay arrays. The array size is computed from the decay memory limit configuration parameter, which is expressed either as an array size or as a maximum hold time ⏳
The decay array size must be big enough to hold the specified decay memory given the time granularity. It also needs to be big enough to hold the number of array elements until integer rounding produces a zero result, if that value is smaller, or else an implementation-imposed reasonable size that stops configurations from using excessive memory. Implementations may choose to make the array size shorter and multiply more than once when decaying a long time interval, to reduce storage 🧠
The reuse index arrays serve a similar purpose to the decay arrays. In BGP, a route is said to be "used" if it's considered the best route. In this context, if the route is "used," it's placed in the RIB and is eligible for advertisement to BGP peers 📣 If a route is withdrawn (a BGP announcement from a peer saying it's no longer reachable), then it's no longer eligible for "use" 💀 When a route becomes reachable, it may not be "used" right away if the figure of merit shows that a recent instability happened. After the route stays stable and the figure of merit decays below the "reuse" threshold, the route is eligible to be "reused," which means it's treated as truly reachable, placed in the RIB, and advertised to peers 🌸 The amount of time until a route can be reused can be found with an array lookup. The array can be built given the decay rate. The array is indexed using a scaled integer that's proportional to the ratio between a current stability figure of merit value and the value needed for the route to be reused 🔥
4.4.3 Per Route State
Information must be maintained per some tuple representing a route. At the very minimum, the NLRI (BGP prefix and length) must be contained in the tuple. Different BGP attributes may be included or excluded depending on the situation. The AS path should also be contained in the tuple by default. The tuple may also optionally contain other BGP attributes, such as MULTI_EXIT_DISCRIMINATOR (MED) 🎀
The tuple representing a route for the purpose of route flap damping is:
tuple entry default options
-------------------------------------------
NLRI
prefix required
length required
AS path included option to exclude
last AS set in path excluded option to include
next hop excluded option to include
MED excluded option to include
in comparisons onlyThe AS path is generally included so you can spot downstream instability that isn't being damped, or isn't being damped enough, and is flip-flopping between a stable and an unstable path 👀 Under rare circumstances it may be desirable to exclude the AS path for all or a subset of prefixes. If an AS path ends in an AS set, in practice the path is always for an aggregate. Changes to the trailing AS set should be ignored. Ideally, the AS path comparison should make sure at least one AS has stayed constant in the old and new AS set, but completely ignoring the contents of a trailing AS set is also acceptable 🧩
Including next hop and MED changes can help suppress the use of an AS that's internally unstable, or avoid a next hop that's closer to an unstable IGP path in the adjacent AS 🌐 If a large number of MED values are used, the increase in state can become a problem. For this reason, MED is disabled by default and only enabled as part of the tuple comparison, using a single state entry regardless of MED value 💜 Including MED will suppress the use of the adjacent AS even though the change doesn't need to be propagated further. Using MED is only a safe practice if a path is known to exist through another AS, or if there are enough peering sites with the adjacent AS that routes heard at only a subset of them will still be suppressed 🫶
4.4.4 Data Structures per Route
The following information must be maintained per route. A route here is considered to be a tuple usually containing NLRI, next hop, and AS path, as defined in Section 4.4.3 ✨
- stability figure of merit (
figure-of-merit)
Each route must have a stability figure of merit per applicable parameter set.
- last time updated (
time-update)
The exact last time updated must be maintained so that exponential decay of the accumulated figure of merit can be deferred until the route might reasonably be considered eligible for a change in status (having gone from unreachable to reachable, or advancing within the reuse lists) 🕰️
- config block pointer
Any implementation that supports multiple parameter sets must provide a way to quickly identify which set of parameters corresponds to the route currently being considered. For implementations supporting only parameter sets where all routes must be treated the same, this pointer is not required 🧚
- reuse list traversal pointers
If doubly linked lists are used to implement reuse lists, then two pointers will be needed, previous and next. Generally there's a double linked list that's unused when a route is suppressed from use, and that can be used for reuse list traversal, which removes the need for additional pointer storage 🦋
4.5 Processing Configuration Parameters 💅
okay so once you've got your configuration parameters, you can precompute a bunch of values you'll keep reusing. saving them up front makes the later calculations way faster 🫶
scaling usually depends on the highest value the figure-of-merit can reach, which we call the ceiling 🔝 the real number value of the ceiling is typically determined by this equation (or you can just configure the ceiling to a specific value, and that then sets T-hold for you ✨):
ceiling = reuse * (exp(T-hold/decay-half-life) * log(2))in that equation, reuse is the reuse threshold from Section 4.2 🎀
the methods for scaled integer arithmetic aren't covered in detail here. we're only going over how to get the real values. turning them into scaled integers and doing the scaled integer math is left up to each implementation 🌸
the ceiling value can be set to the largest integer that fits in half the bits available for an unsigned integer. this lets you multiply the scaled integers by the scaled decay value and then shift them down 💖 implementations can use real numbers or any integer scaling that suits their architecture, whatever vibes best with their hardware 🦋
penalty value and thresholds (as proportional scaled integers)
the figure-of-merit penalty for one route withdrawal, and the cutoff values, have to be scaled using the scaling factor above 🍓
decay rate per tick (decay[1])
next, you need the decay value for each increment of time, as defined by the time granularity. figure this out first as a floating point number if that's easier ✨ the per tick decay is a number just under one. it's the Nth root of one half, where N is the half life divided by the time granularity 🌈
decay[1] = exp ((1 / (decay-half-life/delta-t)) * log (1/2))decay array size (decay-array-size)
the decay array size is the decay memory divided by the time granularity 🧚 if integer truncation drives an array element down to zero, you can make the array smaller. an implementation should also set a sensible maximum array size, or allow more than one multiplication per decay operation 🙈
decay-array-size = (Tmax/delta-t)decay array (decay[])
each i-th element of the decay array is the per tick decay raised to the i-th power 💜 the easiest way is successive floating point multiplies, followed by scaling and integer rounding or truncation. the array only needs to be computed once, at startup 🌟
decay[i] = decay[1] ** i4.6 Building the Reuse Index Arrays 🦋
the reuse lists can get hit pretty hard if a lot of routes are flapping enough to be suppressed 😭 so there's a suggested method for speeding up how you figure out which reuse list to use for a given route. it's introduced in Section 4.2, configured in Section 4.4.2, and the algorithms live in Section 4.8.6 and Section 4.8.7. this section is all about building those reuse list index arrays 🔥
the basis for the array lookup is the ratio of the route's figure-of-merit to the cutoff value. that ratio gets scaled, truncated to an integer, and used as the index into the array. the array entry is an integer that tells you which reuse list to use 👑
reuse array maximum ratio (max-ratio)
this is the maximum ratio between the current stability figure-of-merit and the target reuse value that the reuse array can index. it might be limited by the ceiling set by the maximum hold time, or by how much time the reuse lists cover 💅
max-ratio = min(ceiling/reuse, exp((1 / (half-life/reuse-
array-time)) * log(2)))reuse array scale factor (scale-factor)
the reuse array is an estimator, so its scale factor has to be computed in a way that uses the full size of the reuse array 🎀
scale-factor = reuse-index-array-size / (max-ratio - 1)reuse index array (reuse-index-array[])
each entry in the reuse index array should hold an index into the reuse list array, pointing at one of the list heads. that index should match the reuse list that gets evaluated right after a route becomes eligible for reuse, based on the ratio of the current stability figure-of-merit to the target reuse value for that array entry ✨
reuse-index-array[j] = integer((decay-half-life / reuse-
time-granularity) * log(1/(reuse * (1 + (j / scale-factor)))) /
log(1/2))so here's how you pick the reuse queue for a suppressed route 🧚 divide the current figure-of-merit by the cutoff, subtract one, then multiply by the scale factor. that's your index into the reuse index array (reuse-index-array[]). the value you pull out of there is an index into the array of reuse lists (reuse-array[]). if that index is off the end of the array, use the last queue. otherwise, look in the array and pick the queue at that index 🌈 it's super fast, and the setup and storage totally pay off 💖
4.7 A Sample Configuration 🌸
here's a simple example that estimates the space overhead for a set of configuration parameters 🍓 the design here assumes:
- there is a single parameter set used for all routes,
- decay time for unreachable routes is slower than for reachable routes,
- the arrays must be full size, rather than allowing more than one multiply per decay operation to shrink the array size.
this example gets reused in later sections 🫶 using multiple parameter sets makes things a bit messier. when a route is allowed more than one parameter set, the decay portion of the algorithm gets repeated for each set. if different routes can have different parameter sets, the routes need pointers to their parameter sets so finding them stays quick, but the algorithms are otherwise the same 🦋
a sample set of configuration parameters and a sample set of implementation parameters are in the two lists below 👑
- Configuration Parameters
cut = 1.25reuse = 0.5T-hold = 15 minsdecay-ok = 5 mindecay-ng = 15 minTmax-ok, Tmax-ng = 15, 30 mins
- Implementation Parameters
delta-t = 1 secdelta-reuse = 15 secreuse-list-size = 256reuse-index-array-size = 1,024
using these parameters and the equations from Section 4.5, you can work out the space overhead 💅 there's a fixed overhead that doesn't depend on how many routes you have. then there's the space needed for a stable route, and a bigger chunk for an unstable route 😭 the space requirements for the parameters above are listed below ✨
- fixed overhead (using parameters from previous example)
900 * integer - decay array1,800 * integer - decay array120 * pointer - reuse list-heads2,048 * integer - reuse index arrays
- overhead per stable route
pointer - containing null entry
- overhead per unstable route
pointer - to a damping structure containing the followinginteger - figure of merit + bit for stateinteger - last time updated2 * pointer - reuse list pointers (prev, next)
the decay arrays are sized according to delta-t and Tmax-ok or Tmax-ng 🧚 the number of reuse list-heads depends on delta-reuse and the greater of Tmax-ok or Tmax-ng 🌈 there are two reuse index arrays, and their size is a configured parameter 💜
figure 3 shows how the algorithm behaves with the parameters above 🎀 there are four cases. in all four, the route oscillates for a twelve minute period. two oscillation periods are tested, 2 minutes and 4 minutes, and two duty cycles, one where the route is reachable for 20% of the cycle and one where it's reachable for 80% 😭 in all four cases, the route gets suppressed after it becomes unreachable the second time. once it's suppressed, it stays that way until some time after it becomes stable 💀
the routes that oscillate on a 4 minute period stop being suppressed within 9-11 minutes after becoming stable ✨ the routes with a 2 minute period stay suppressed for nearly the maximum 15 minute period after becoming stable 🔥
the vibes are immaculate, and we're only getting started 🫶
4.8 Processing Routing Protocol Activity ✨
okay so the earlier sections are all about config params, how they map to the run-time stuff, and how to set up storage 🎀 this section is the actual what happens when things go wrong part, aka how the router reacts to routing events and timer events while it's running 🔥
The routing events are:
- A BGP peer or new route comes up for the first time (or after an extended down time) (Section 4.8.1)
- A route becomes unreachable (Section 4.8.2)
- A route becomes reachable again (Section 4.8.3)
- A route changes (Section 4.8.4)
- A peer goes down (Section 4.8.5)
time figure-of-merit as a function of time (in minutes)
0.00 0.000 . 0.000 . 0.000 . 0.000 .
0.62 0.000 . 0.000 . 0.000 . 0.000 .
1.25 0.000 . 0.000 . 0.000 . 0.000 .
1.88 0.000 . 0.000 . 0.000 . 0.000 .
2.50 0.977 . 0.968 . 0.000 . 0.000 .
3.12 0.949 . 0.888 . 0.000 . 0.000 .
3.75 0.910 . 0.814 . 0.000 . 0.000 .
4.37 1.846 . 1.756 . 0.983 . 0.983 .
5.00 1.794 . 1.614 . 0.955 . 0.935 .
5.63 1.735 . 1.480 . 0.928 . 0.858 .
6.25 2.619 . 2.379 . 0.901 . 0.786 .
6.88 2.544 . 2.207 . 0.876 . 0.721 .
7.50 2.472 . 2.024 . 0.825 . 0.661 .
8.13 3.308 . 2.875 . 1.761 . 1.608 .
8.75 3.213 . 2.698 . 1.711 . 1.562 .
9.38 3.122 . 2.474 . 1.662 . 1.436 .
10.00 3.922 . 3.273 . 1.615 . 1.317 .
10.63 3.810 . 3.107 . 1.569 . 1.207 .
11.25 3.702 . 2.849 . 1.513 . 1.107 .
11.88 3.498 . 2.613 . 1.388 . 1.015 .
12.50 3.904 . 3.451 . 2.312 . 1.953 .
13.13 3.580 . 3.164 . 2.120 . 1.791 .
13.75 3.283 . 2.902 . 1.944 . 1.643 .
14.38 3.010 . 2.661 . 1.783 . 1.506 .
15.00 2.761 . 2.440 . 1.635 . 1.381 .
15.63 2.532 . 2.238 . 1.499 . 1.267 .
16.25 2.321 . 2.052 . 1.375 . 1.161 .
16.88 2.129 . 1.882 . 1.261 . 1.065 .
17.50 1.952 . 1.725 . 1.156 . 0.977 .
18.12 1.790 . 1.582 . 1.060 . 0.896 .
18.75 1.641 . 1.451 . 0.972 . 0.821 .
19.38 1.505 . 1.331 . 0.891 . 0.753 .
20.00 1.380 . 1.220 . 0.817 . 0.691 .
20.62 1.266 . 1.119 . 0.750 . 0.633 .
21.25 1.161 . 1.026 . 0.687 . 0.581 .
21.87 1.064 . 0.941 . 0.630 . 0.533 .
22.50 0.976 . 0.863 . 0.578 . 0.488 .
23.12 0.895 . 0.791 . 0.530 . 0.448 .
23.75 0.821 . 0.725 . 0.486 . 0.411 .
24.37 0.753 . 0.665 . 0.446 . 0.377 .
25.00 0.690 . 0.610 . 0.409 . 0.345 .Figure 3: Some fairly long route flap cycles, repeated for 12 minutes, followed by a period of stability.
the reuse list is basically a fast-lane for routes that got suppressed 🏃♀️💨 they either got stable long enough to be reused, or they got suppressed for so long that they can be treated like a brand new route 🌸 two operations are described:
- Inserting into a reuse list (Section 4.8.6)
- Reuse list processing every delta-t seconds (Section 4.8.7)
4.8.1 Processing a New Peer or New Routes
when a peer comes up, there's nothing to do if the routes have no previous history of instability 🧚 like if this is the first time the peer is coming up and announcing these routes. for each route, the pointer to the damping structure gets zeroed and the route gets used. same vibe for a new route, or a route that's been down long enough that its figure of merit hit zero and the damping structure got deleted 💅
4.8.2 Processing Unreachable Messages
when a route is withdrawn or changed (Section 4.8.4 explains how a change is handled), this procedure runs 🦋
if there's no previous stability history (the damping structure pointer is zero), then:
- allocate a damping structure
- set figure-of-merit = 1
- withdraw the route
otherwise, if there's an existing damping structure, then:
1. set t-diff = t-now - t-updated
2. if (t-diff puts you off the end of the array) {
setfigure-of-merit =1
}else {
setfigure-of-merit =figure-of-merit *decay-array-ok [t-diff ]+ 1
if(figure-of-merit >ceiling) {
setfigure-of-merit =ceiling
}
}
3. remove the route from a reuse list if it is on one
4. withdraw the route unless it is already suppressedin either case, then:
- set t-updated = t-now
- insert into a reuse list (see Section 4.8.6)
if there was a stability history, the previous value of the stability figure of merit gets decayed 🫶 this uses the decay array (decay-array). the index comes from subtracting the last time updated from the current time, then dividing by the time granularity. if the index is zero, the figure of merit stays the same (no decay). if it's bigger than the array size, it gets zeroed. otherwise, use the index to grab a decay array element and multiply the figure of merit by that element. if you're using the suggested scaled integer method, shift down half an integer. then add the scaled penalty for one more unreachable (shown above as 1). if the result is above the ceiling, swap it for the ceiling value. last step: update the last time updated field, ideally taking into account how much time got truncated before the decay calc 💖
when a route becomes unreachable, alternate paths must be considered 👑 this gets a little messy if different config params are used depending on whether there are viable alternate paths or not. if all of these alternate paths got suppressed because there used to be an alternate route, and the withdrawal changes that situation, the suppressed alternate paths must be reevaluated. they should be reevaluated in order of normal route preference 🌈 once you hit one of these alternate routes that was suppressed but is now usable (since there's no alternate route anymore), you don't need to reevaluate any more routes. this only applies if routes get two different reuse thresholds: one for when there's an alternate path, and a higher one for when suppressing the route would make the destination completely unreachable 💀
4.8.3 Processing Route Advertisements
when a route gets readvertised and there's no damping structure, the procedure is the same as in Section 4.8.1:
- don't create a new damping structure
- use the route
if a damping structure exists, the figure of merit gets decayed, and the figure of merit and last time updated fields get updated 🍓 then a decision gets made: can the route be used right away, or does it need to be suppressed for a while?
1. set t-diff = t-now - t-updated
2. if (t-diff puts you off the end of the array) {
set figure-of-merit =0
}else {
set figure-of-merit= figure-of-merit* decay-array-ng[t-diff]
}
3. if ( not suppressed and figure-of-merit < cut ) {
use the route
}else if( suppressed and figure-of-merit< reuse) {
set state tonot suppressed
remove the route from a reuse list
use the route
}else {
set state to suppressed
don't use the route
insert into a reuse list (see Section 4.8.6)
}
4. if ( figure-of-merit > 0 ) {
set t-updated= t-now
}else {
recover memory for damping struct
zero pointer to damping struct
}if the route is deemed usable, a search for the current best route must be made 🔍 the newly reachable route then gets evaluated against the BGP protocol rules for route selection.
if the new route is usable, the previous best route gets examined. before the route comparisons, the current best route may have to be reevaluated if separate param sets are used depending on whether there's an alternate route or not 🧐 if there had been no alternate, the previous best route may get suppressed.
if the new route is going to be suppressed, it only gets placed on a reuse list if it would have been preferred over the current best route had the new route been accepted as stable 🙈 there's no point queuing a route on a reuse list if, after it becomes usable, it wouldn't be used anyway because a more preferred route exists. a route like that wouldn't need to be reevaluated unless the preferred route became unreachable. as written here, the less preferred route would get reevaluated, and maybe used or maybe added to a reuse list, when the withdrawal of a more preferred best route gets processed. ngl that's a lot of extra work 😭
4.8.4 Processing Route Changes
if a peer router replaces a route by supplying a new path, the route being replaced should be treated as if an unreachable had been received (see Section 4.8.2) ✨ this happens when a peer somewhere back in the AS path keeps switching between two AS paths and isn't damping route flap (or is applying less damping). there's no way to tell if one AS path is stable and the other is flapping, or if they're both flapping 🌀 if the cycle is short compared to convergence times, neither route through that peer will deliver packets very reliably. since there's no way to affect that peer so it picks the stable one of the two AS paths, the only viable option is to penalize both routes by treating each change as an unreachable followed by a route advertisement 🎀
4.8.5 Processing A Peer Router Loss
when a peer routing session breaks, either all the individual routes advertised by that peer may be marked as unstable, or the peering session itself may be marked as unstable 🦋 marking the peer saves a lot of memory. since the individual routes get advertised as unreachable to routers beyond the immediate problem, per-route state will be incurred beyond the peer that's immediately adjacent to the BGP session that went down. if the instability keeps going, the immediately adjacent router only needs to keep track of the peer's stability history 🧚 the routers further out won't get any more advertisements or withdrawals of routes, and they'll dispose of the damping structure over time.
BGP notification through an optional transitive attribute that damping will already be applied may be considered in the future to reduce the number of routers that incur damping structure storage overhead 🌸
4.8.6 Inserting into the Reuse Timer List 🧚
Okay so the reuse lists are how we quickly check on routes that got suppressed but then stayed stable long enough to be reused again. 💅 It's a series of list heads, and each list holds the routes that are scheduled for reevaluation at roughly the same time. The list heads are treated as a circular array, so picture a ring of queues. See Figure 4. 🦋
A simple way to build that ring is an array holding the list heads, plus an offset that is used whenever you access the array. The offset tells you which list is the first one. The Nth list sits at the index you get from N plus the offset, taken modulo the number of list heads. The examples that follow all assume this design. ✨
The key requirement is being able to drop an entry into the most appropriate queue with minimal computation, and the only input you get is the current figure-of-merit. 😭 So instead of doing a logarithm, we use the reuse array (reuse-array[]) from Section 4.6. The array, the scale and the bounds are all precomputed so they can map a figure-of-merit to the nearest list head with no logarithm needed (see Section 4.5). Iconic. 👑
+-+ +-+ +-+ non-empty linked list means
| | | | | | <-- that there are routes with
+-+ +-+ +-+ defered action to be taken
^ ^ ^ N * delta-reuse seconds later.
| | |
+------+------+------+------+------+ +------+
| list | list | list | list | list | ... | list |
| head | head | head | head | head | ... | head |
+------+------+------+------+------+ +------+
^ ^ ^ ^ ^ ^
Nth 1st 2nd 3rd 4th N-1
|
offset to first list
(the offset is incremented every delta-reuse seconds)
Figure 4: Reuse List Data StructuresQuick heads up on notation, because it's a little weird 🙈: in the next bits, the operator prefix "modulo a b" means "b % a" in C-style algebra. So modulo 16 1023 comes out to 15. Easy once you know. 🍓
Here's the insertion procedure, step by step:
1. scale figure-of-merit for the index array lookup producing
index
2. check index against the array bound
3. if (within the array bound) {
set index =reuse-array [index ]
}else {
set index =reuse-list-size -1
}
4. insert into the list
reuse-list[ moduloreuse-list-size (index +offset )]Picking the right reuse list is honestly lowkey simple. You do a multiply and a shift to scale, take the integer truncation, then do one array lookup in the reuse array (reuse-array[]). Whatever comes back is the reuse list you use. 🌸 The reuse list is a circular list, and the usual trick is an array plus an offset and a modulo to pick the right entry. Then you just increment the offset to rotate the whole ring. Main character energy for sure. 💜
4.8.7 Handling Reuse Timer Events ⏰
So the reuse timer should be coarser than the decay timer. 🙈 That means when the reuse timer fires, suppressed routes need to be decayed by several increments of the decay time at once. To skip some computation, you can always insert into the reuse list for one time increment past reuse eligibility. ✨ And if the reuse lists have a longer "memory" than the "decay memory" (described above), then everything in the first queue is ready for immediate reuse if it's reachable. If it's unreachable, the history entry can just get disposed of. Periodt. 💅
When it's time to move the lists forward, you have to process the first queue on the reuse list and then rotate the circular queue. Using an array and an offset as a circular array (as described in Section 4.8.6), the algorithm below runs every delta-reuse seconds. 🦋
1. save a pointer to the current zeroth queue head and zero the
list head entry
2. set offset = modulo reuse-list-size ( offset + 1 ), thereby
rotating the circular queue of list-heads
3. if ( the saved list head pointer is non-empty )
for each entry {
sett-diff =t-now -t-updated
set figure-of-merit =figure-of-merit *decay-array-ok [t-diff ]
sett-updated =t-now
if( figure-of-merit< reuse)
reuse the route
else
re-insert into another list (seeSection 4.8.6)
}Here's the vibe. The zeroth list head gets saved and its array entry gets zeroed. Then the list heads advance by bumping the offset. Starting from the saved head of the old zeroth list, each route gets reevaluated, and then it's either used, thrown away entirely, or requeued if it isn't ready for reuse yet. 😭 And if a route does get used, it has to be treated like a brand new route advertisement, same as in Section 4.8.3. Ate. 🔥
5 Implementation Experience 💅
Okay, the very first implementations of "route flap damping" were the route server daemon (rsd), written by Ramesh Govindan (ISI), and the Cisco IOS implementation by Ravi Chandra. 🎀 Both showed up in 1995 and have been used a ton since. The rsd version has been running on route servers at the NSF funded Network Access Points (NAPs) and at other major Internet interconnects. ✨ The Cisco IOS version has been used by Internet Service Providers all over the world. 🌈 The rsd code got folded into releases of gated (see http://www.gated.org), and it's available in commercial routers that run gated. Iconic lineage, honestly. 👑
There are now more than 2 years of BGP route damping deployment experience under the belt. 🧚 Some problems have shown up in the wild, but so far they're all solvable with careful implementation of the algorithm and careful deployment. In some topologies, coordinated deployment can really help 🫶, and in every case, being open about using route damping and about the parameters you chose is super beneficial when you're debugging connectivity problems. 🙈
Some of the problems came from subtle implementation errors. Route damping should never be applied on IBGP learned routes 🚫. Doing it can open the door to persistent route loops. When IBGP routes inside an AS are inconsistent, route loops can easily form, and suppressing IBGP learned routes causes exactly that kind of inconsistency. So implementations should disallow configuring route damping on IBGP peers. Not it. 💀
Penalties for instability should only be applied when a route is removed or replaced, and not when a route is added. 🌸 If the damping parameters are applied consistently, this constraint means a stable secondary path gets preferred over an unstable primary path, because the primary got damped near the source. Lowkey genius. 💖
In topologies where several AS paths lead to the same destination, flapping on the primary path can end up suppressing the secondary path. 😭 That happens when no damping is happening near the cause of the flap, or when a distant AS is damping more aggressively. There are two ways to fix it. One is to damp near the source of the flap and keep the damping parameters consistent. The other is for a distant AS that insists on more aggressive damping to turn off penalizing routes on AS path change, so it only penalizes routes that get withdrawn completely. To do that, the implementation has to support this option (as described in Section 4.4.3). 🦋
Route flap should be damped near the source. Single homed destinations can be covered by static routes, and aggregation is another way to damp. 🌷 Providers should damp their own internal problems, though damping on IGP link state origination isn't implemented by router vendors yet. Providers that use multiple AS inside their own topology should damp between their own AS. And providers should damp adjacent providers' AS. 🧚
Damping limits how much excessive route change gets propagated when connectivity is super intermittent. 💜 Once a problem is fixed, the damping state for the prefixes known to be damped because of that problem can be cleared by hand. To figure out where damping might have happened after connectivity problems, providers should publish their damping parameters. ✨ And providers should be willing to manually clear damping on specific prefixes or AS paths when another provider asks, as long as the request comes with credible assurance that the problem has really been addressed. 🫶
By damping their own routing information, providers can cut down on how often they need to ask other providers to clear damping state after fixing a problem. 🎀 Providers should be proactive and keep an eye on which prefixes and paths are suppressed, in addition to monitoring link states and BGP session state. Stay on top of it, bestie. 👑
Acknowledgements 🫶
This work and this document might not have happened without the advice, comments and encouragement of Yakov Rekhter (Cisco). 💖 Dennis Ferguson (MCI) described the algorithms in the gated BGP implementation and offered lots of valuable comments and insights. David Bolen (ANS) and Jordan Becker (ANS) gave valuable comments, especially about the early simulations. 🌈 More than four years passed between the initial draft presented to the BGP WG (October 1993) and this iteration. At the time of writing there's already significant experience with two implementations, each deployed since 1995. One was led by Ramesh Govindan (ISI) for the NSF Routing Arbiter project, and the second was led by Ravi Chandra (Cisco). ✨ Sean Doran (Sprintlink) and Serpil Bayraktar (ANS) were among the early independent testers of the Cisco pre-beta implementation. Valuable comments and implementation feedback came from many people on the IETF IDR WG and the RIPE Routing Work Group, plus NANOG and IEPG. 🧚
Big thanks also to Rob Coltun (Fore Systems), Sanjay Wadhwa (Fore), John Scudder (IENG), Eric Bennet (IENG) and Jayesh Bhatt (Bay Networks) for catching errors in the math while coding more recent implementations. 🔥 Those errors were in the details of the implementation suggestion sections, which got written after the first two implementations were already done. And thank you to Vern Paxson for a super thorough review that led to tons of clarifications in the document. Literally a legend. 🌟
References ✨
Okay bestie, here's the reading list 📚💅 every entry is still exactly as the RFC lists it:
- [1] Gross, P., and Y. Rekhter, "Application of the border gateway protocol in the internet", RFC 1268, October 1991.
- [2] ISO/IEC. Iso/iec 10747 - information technology - telecommunications and information exchange between systems - protocol for exchange of inter-domain routeing information among intermediate systems to support forwarding of iso 8473 pdus. Technical report, International Organization for Standardization, August 1994.
ftp://merit.edu/pub/iso/idrp.ps.gz. - [3] Lougheed, K., and Y. Rekhter, "A border gateway protocol 3 (BGP-3)", RFC 1267, October 1991.
- [4] Rekhter, Y., and P. Gross, "Application of the border gateway protocol in the internet", RFC 1772, March 1995.
- [5] Rekhter, Y., and T. Li, "A border gateway protocol 4 (BGP-4)", RFC 1771, March 1995.
- [6] Rekhter, Y., and C. Topolcic,"Exchanging routing information across provider boundaries in the CIDR environment", RFC 1520, September 1993.
- [7] Traina, P., "BGP-4 protocol analysis", RFC 1774, March 1995.
- [8] Traina, P., "Experience with the BGP-4 protocol", RFC 1773, March 1995.
Security Considerations 🔒
Okay, this part is serious, so pay attention 🫶 The practices in this document do not make the security of the routing protocols any weaker. Yes, denial of service is possible in a routing environment that's already insecure, but these practices only add to how long those attacks can keep going. They don't change how you prevent attacks, and they don't change how you figure out where an attack is coming from. Lowkey important stuff 💀
Authors' Addresses 🎀
- Curtis Villamizar, ANS, EMail: curtis@ans.net
- Ravi Chandra, Cisco Systems, EMail: rchandra@cisco.com
- Ramesh Govindan, ISI, EMail: govindan@isi.edu
Full Copyright Statement 💜
Copyright (C) The Internet Society (1998). All Rights Reserved. 🌈
So here's the deal with copying this doc 📄 You can copy it and hand it to people, and you can write derivative works that comment on it, explain it, or help people implement it. You can publish and distribute those too, in whole or in part, with no restrictions, as long as the copyright notice and this paragraph stay on every copy and derivative work. The one thing you can't do is modify this document itself, like removing the copyright notice or the references to the Internet Society or other Internet organizations. The exceptions are when you're developing Internet standards, where the Internet Standards process copyright procedures have to be followed, or when you're translating it into a language other than English. ✨
The permissions above are perpetual, and the Internet Society or its successors or assigns will never revoke them. 🦋
This document and the information in it are provided on an "AS IS" basis, and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 🙈
📦 where this page comes from
- ietf: rfc2439 1998-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms