RFC 9494

Long-Lived Graceful Restart for BGP 💖

okay bestie, here's the title block tea 🫖✨ This is RFC 9494, a Standards Track doc from the IETF (Internet Engineering Task Force), so it's the real deal 💅

• Authors: J. Uttaro (Independent Contributor), E. Chen (Palo Alto Networks), B. Decraene (Orange), J. Scudder (Juniper Networks)
• Date: November 2023
• Category: Standards Track
• ISSN: 2070-1721
• Updates: 6368 (it patches RFC 6368, more on that in the Abstract 👀)
• Obsoletes: nothing listed in the title block

Abstract

So this doc introduces a BGP capability called the "Long-Lived Graceful Restart Capability" (aka the "LLGR Capability") 🦋 The big perk is that stale routes can hang around for way longer after a session dies than BGP Graceful Restart (from RFC 4724) lets them. There's a well-known BGP community called "LLGR_STALE" for marking those long-lived stale routes 🏷️ and a second one called "NO_LLGR" for marking routes that these procedures should NOT be applied to 🙅‍♀️

Long-lived stale routes get treated as the least preferred 📉 and only get advertised to BGP speakers that also advertised the capability. Also, this extension isn't advisable in every situation, so the doc gives some guidelines to help you figure out if it's a good fit 🧠

This memo updates RFC 6368 by saying that the LLGR_STALE community must be propagated into, or out of, the path attributes exchanged between the Provider Edge (PE) and Customer Edge (CE) routers. Main character energy 👑

Status of This Memo

This is an Internet Standards Track document.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further info on Internet Standards is in Section 2 of RFC 7841 📚

Info about the current status of this document, any errata, and how to give feedback on it lives at https://www.rfc-editor.org/info/rfc9494 🔗

Copyright Notice

Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. 🎀

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this doc. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Revised BSD License ⚖️

Table of Contents

   1.  Introduction
   2.  Terminology
     2.1.  Definitions
     2.2.  Abbreviations
     2.3.  Requirements Language
   3.  Protocol Extensions
     3.1.  Long-Lived Graceful Restart Capability
     3.2.  LLGR_STALE Community
     3.3.  NO_LLGR Community
   4.  Theory of Operation
     4.1.  Use of the Graceful Restart Capability
     4.2.  Session Resets
     4.3.  Processing LLGR_STALE Routes
     4.4.  Route Selection
     4.5.  Errors
     4.6.  Optional Partial Deployment Procedure
     4.7.  Procedures When BGP Is the PE-CE Protocol in a VPN
       4.7.1.  Procedures When EBGP Is the PE-CE Protocol in a VPN
       4.7.2.  Procedures When IBGP Is the PE-CE Protocol in a VPN
   5.  Deployment Considerations
     5.1.  When BGP Is the PE-CE Protocol in a VPN
     5.2.  Risks of Depreferencing Routes
   6.  Security Considerations
   7.  Examples of Operation
   8.  IANA Considerations
   9.  References
     9.1.  Normative References
     9.2.  Informative References
   Acknowledgements
   Contributors
   Authors' Addresses

1. Introduction ✨

Routing protocols in general, and BGP especially, have historically been designed around "correctness" 🎯 A big part of correctness is that each network element's forwarding state converges to the current state of the network as fast as possible. So the protocol was built to remove state advertised by routers that went down (from a BGP perspective) as quickly as it could. That's been loosened a bit over time, most notably by BGP Graceful Restart (GR) [RFC4724], but the vibe has stayed the same: rapidly clear out stale state from the network 🧹

Two things have shown up over time that make us rethink the assumptions behind that paradigm 🤔

1. The widespread adoption of tunneled forwarding infrastructures (for example, MPLS). These eliminate the risk of some forwarding loops that can happen in hop-by-hop forwarding, so they reduce one of the reasons forwarding elements needed strong consistency with each other.
2. The increasing use of BGP as a transport for data that's less closely tied to packet forwarding than it originally was. Examples include BGP for auto-discovery (Virtual Private LAN Service (VPLS) [RFC4761]) and filter programming (Flow Specification (FLOWSPEC) [RFC8955]). In these cases, BGP data acts more like configuration than conventional routing ⚙️

All of this means network operators might want to keep BGP data around for longer than has been possible so far, when the BGP control plane fails for some reason 💭 BGP Graceful Restart [RFC4724] comes close to what's wanted, but there are gaps, most notably in how long stale info can be kept: Graceful Restart has a 4095-second upper bound ⏱️

So in this document, we introduce a BGP capability called the "Long-Lived Graceful Restart Capability." The goal is that stale information can be retained for a longer time across a session reset 🦋 We also introduce two BGP well-known communities:

• LLGR_STALE, to mark such information, and
• NO_LLGR, to indicate that these procedures should not be applied to the marked route.

Long-lived stale information is treated as least preferred, and its advertisement is limited to BGP speakers that support the capability. Where possible, we reference the semantics of BGP Graceful Restart [RFC4724] instead of re-specifying similar semantics in this doc 📖

The expected deployment model is that this extension will only be invoked for certain address families. Section 5 goes into more detail on this. The use of this extension may be combined with conventional Graceful Restart. If it is, LLGR gets invoked after the conventional Graceful Restart interval has elapsed. When it's not combined, LLGR is invoked immediately 🚀

Apart from the potential to greatly extend the timer, the most obvious difference between LLGR and conventional Graceful Restart is that in LLGR, routes are "depreferenced". That means they're treated as least preferred. In conventional GR, route preference isn't affected 💜

The design choice to treat long-lived stale routes as least preferred came from the expectation that they might be retained for (potentially) an almost unbounded period of time. In conventional Graceful Restart, stale routes are only kept for a brief interval ⏳ For Graceful Restart, the trade-off between advertising new route status (at the cost of routing churn) and not advertising it (at the cost of suboptimal or incorrect route selection) gets resolved in favor of not advertising. For LLGR, it gets resolved in favor of advertising new state, and stale info is only used as a last resort 🫶

Section 7 has some simple examples showing how this extension works 🧚

2. Terminology 📝

2.1. Definitions

• Depreference: A route is said to be depreferenced if it has its route selection preference reduced in reaction to some event.
• Helper: Sometimes called a "helper router." During Graceful Restart or Long-Lived Graceful Restart, it's the router that detects a session failure and applies the listed procedures. [RFC4724] calls this the "receiving speaker" 🦸‍♀️
• Route: In this document, "route" means any information encoded as BGP Network Layer Reachability Information (NLRI) and a set of path attributes. As discussed above, the connection between such routes and the installation of forwarding state may be pretty remote 🌸

Further note that, for brevity, when this document references conventional Graceful Restart, it cites its base specification, [RFC4724]. That specification has been updated by [RFC8538]. The citation to [RFC4724] isn't meant to be limiting 💅

2.2. Abbreviations

• CE: Customer Edge (See [RFC4364] for more information on Customer Edge routers.)
• EoR: End-of-RIB (See Section 2 of [RFC4724] for more information on End-of-RIB markers.)
• GR: Graceful Restart (See [RFC4724] for more information on GR.) This term is also sometimes called "conventional Graceful Restart" or "conventional GR," to tell it apart from the "Long-Lived Graceful Restart" or "LLGR" defined by this document 🍓
• LLGR: Long-Lived Graceful Restart
• LLST: Long-Lived Stale Time
• PE: Provider Edge (See [RFC4364] for more information on Provider Edge routers.)
• VRF: VPN Routing and Forwarding (See [RFC4364] for more information on VRF tables.)

2.3. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🎀 Lowercase versions are just regular words, bestie 🙈

3. Protocol Extensions 🦋

A BGP capability and two BGP communities are introduced in the subsections that follow. Let's go! 🚀

3.1. Long-Lived Graceful Restart Capability ✨

Okay so the Long-Lived Graceful Restart Capability (aka the "LLGR Capability") is a BGP capability with value 71 🦋 It's how a BGP speaker says "hey, I can hold onto my state through a restart, bestie" 💖 It's a BGP capability [RFC5492], and if you're advertising LLGR, you MUST also advertise the Graceful Restart capability [RFC4724] 🫶 (more on that in Section 4.1, obsessed with the structure tbh 🎀).

The capability value is zero or more tuples of <AFI, SAFI, Flags, LLST>, and it's giving very neat little stacked fields 🌸

+--------------------------------------------------+
| Address Family Identifier (16 bits)              |
+--------------------------------------------------+
| Subsequent Address Family Identifier (8 bits)    |
+--------------------------------------------------+
| Flags for Address Family (8 bits)                |
+--------------------------------------------------+
| Long-Lived Stale Time (24 bits)                  |
+--------------------------------------------------+
| ...                                              |
+--------------------------------------------------+
| Address Family Identifier (16 bits)              |
+--------------------------------------------------+
| Subsequent Address Family Identifier (8 bits)    |
+--------------------------------------------------+
| Flags for Address Family (8 bits)                |
+--------------------------------------------------+
| Long-Lived Stale Time (24 bits)                  |
+--------------------------------------------------+

Here's what each field means 💅

Address Family Identifier (AFI) and Subsequent Address Family Identifier (SAFI): Taken together, these say the speaker can keep its forwarding state for that address family across a later BGP restart ✨ Routes can be either:

• explicitly tied to a particular AFI and SAFI if they use the encoding in [RFC4760], or
• implicitly tied to <AFI=IPv4, SAFI=Unicast> if they use the encoding in [RFC4271] 🌈

Flags for Address Family: This is a bit field for routes advertised with that AFI and SAFI 🔥

                              0 1 2 3 4 5 6 7
                             +-+-+-+-+-+-+-+-+
                             |F|   Reserved  |
                             +-+-+-+-+-+-+-+-+

The most significant bit is the F bit 🦋 When it's set (value 1), it means the state for those routes was actually preserved during the previous BGP restart. It's called the "F bit" because it used to mean "forwarding state preserved" 🧚 Use of the F bit is detailed in Section 4.2. The remaining bits are reserved, so they MUST be set to zero by the sender and ignored by the receiver 💜

Long-Lived Stale Time: This is a time in seconds for how long stale info (for this AFI/SAFI) can be kept by the receiver, on top of the "Restart Time" from the Graceful Restart Capability ⏳ Because use cases for this extension vary wildly, there's no suggested default value for the LLST 🙈

3.2. LLGR_STALE Community ✨

The well-known BGP community LLGR_STALE (value: 0xFFFF0006) can mark stale routes that are kept for a longer time 🌸 (see [RFC1997] if you want the deets on BGP communities). Those long-lived stale routes get handled according to the procedures in Section 4 💖

An implementation MAY let users configure policies that accept, reject, or modify routes based on whether this community is present or absent 🎀

3.3. NO_LLGR Community ✨

The well-known BGP community NO_LLGR (value: 0xFFFF0007) marks routes that a BGP speaker does NOT want treated under these procedures, as Section 4 details 🙅‍♀️

An implementation MAY let users configure policies that accept, reject, or modify routes based on whether this community is present or absent 💅

---

4. Theory of Operation 🔮

If a BGP speaker is set up to support this document, it MUST use BGP Capabilities Advertisement [RFC5492] to advertise the Long-Lived Graceful Restart Capability 🫶 How you set the parameters for an AFI/SAFI depends on the BGP speaker's properties, network scale, and local configuration 🌈

When the Long-Lived Graceful Restart Capability is present, the procedures in [RFC4724] keep applying unless this document explicitly revises them ✨

4.1. Use of the Graceful Restart Capability 🎀

If the LLGR Capability is advertised, the Graceful Restart capability MUST also be advertised. If it isn't, the LLGR Capability MUST be disregarded 🙅‍♀️ The reason for this is to reuse base mechanisms common to both "flavors" 💖 Those are: origination, collection, and processing of EoR, plus the finite-state-machine changes and connection-reset logic that GR introduced 🧚

We observe that if you don't want conventional Graceful Restart on the session, you can skip the conventional GR phase by leaving all AFIs/SAFIs out of the GR Capability, advertising a Restart Time of zero, or doing both 🦋 Section 4.2 covers how conventional GR and LLGR interact 👑

4.2. Session Resets 🌸

[RFC4724] defines the conditions under which a BGP session can reset and keep its associated routes ✨ If that kind of reset happens on a session where the LLGR Capability was also exchanged, these procedures apply 💅

• If the Graceful Restart Capability that was received does not list all AFIs/SAFIs supported by the session, then the GR Restart Time shall be deemed zero for the AFIs/SAFIs that aren't listed 🙈
• Similarly, if the received LLGR Capability does not list all AFIs/SAFIs supported by the session, then the Long-Lived Stale Time shall be deemed zero for the AFIs/SAFIs that aren't listed 🌈

The following text in Section 4.2 of [RFC4724] no longer applies 💀

If the session does not get re-established within the "Restart Time" that the peer advertised previously, the Receiving Speaker MUST delete all the stale routes from the peer that it is retaining.

Instead, these procedures are specified 🫶

After the session goes down, and before it's re-established, the stale routes for an AFI/SAFI MUST be retained 🦋 How long they're kept is limited by the sum of the Restart Time in the received Graceful Restart Capability and the Long-Lived Stale Time in the received Long-Lived Graceful Restart Capability ⏳ The timers received in the Long-Lived Graceful Restart Capability SHOULD be modifiable by local configuration, which may impose an upper bound, a lower bound, or both on their values 🎀

If the Restart Time or the Long-Lived Stale Time is zero, that period lasts zero seconds 🙈 For example, if the Restart Time is zero and the Long-Lived Stale Time is nonzero, only the LLGR procedures apply 💖 If it's the other way around (Long-Lived Stale Time zero, Restart Time nonzero), only the GR procedures apply 🌸 If both are zero, none of these procedures apply, only the base BGP specification [RFC4271] does (although EoR would still be used as detailed in [RFC4724]) 💅 And if both are nonzero, the procedures run serially: GR first, then LLGR 🔥 During the first interval, route preference is not affected while GR procedures are in effect ✨ During the second interval, while LLGR procedures are in effect, routes are treated as least preferred, as specified elsewhere in this document 🧚

Once the Restart Time period ends (including when the Restart Time is zero), the LLGR period is said to have begun, and the following procedures MUST be performed 👑

• For each AFI/SAFI for which a nonzero Long-Lived Stale Time was received, the helper router MUST start a timer for that Long-Lived Stale Time. If that timer expires before the session is re-established, the helper MUST delete all stale routes of that AFI/SAFI from the neighbor that it is retaining 💀
• The helper router MUST attach the LLGR_STALE community to the stale routes it is retaining. Note that this requirement implies the routes would need to be readvertised so the modified community gets disseminated 🦋
• If any routes from the peer have been marked with the NO_LLGR community, either as sent by the peer or as the result of a configured policy, they MUST NOT be retained and MUST be removed as per the normal operation of [RFC4271] 🙅‍♀️
• The helper router MUST perform the procedures listed in Section 4.3 🎀

Once the session is re-established, the procedures in [RFC4724] apply to the stale routes, whether they were retained during the Restart Time period or the Long-Lived Stale Time period 🌈 However, in the case of consecutive restarts, the previously marked stale routes MUST NOT be deleted before the Long-Lived Stale Time timer expires 💜

Similar to [RFC4724], once the LLGR Period begins, the Helper MUST immediately remove all the stale routes from the peer that it is retaining for that address family if any of the following occur 😭

• the F bit for a specific address family is not set in the newly received LLGR Capability, or
• a specific address family is not included in the newly received LLGR Capability, or
• the LLGR and accompanying GR Capability are not received in the re-established session at all.

If a Long-Lived Stale Time timer is running for routes with a given AFI/SAFI received from a peer, it MUST NOT be updated (other than by manual operator intervention) until the peer has established and synchronized a new session 🔒 The session counts as "synchronized" for a given AFI/SAFI once the EoR for that AFI/SAFI has been received from the peer, or once the Selection_Deferral_Timer discussed in [RFC4724] expires 🫶

The value of a Long-Lived Stale Time in the capability received from a neighbor MAY be reduced by local configuration ✨

While the session is down, expiry of a Long-Lived Stale Time timer is treated analogously to expiry of the Restart Time timer in [RFC4724], except that it only applies to the AFI/SAFI it accompanies 🦋 However, the timer keeps running once the session has re-established 🧚 It is neither stopped nor updated until the EoR marker is received for the relevant AFI/SAFI from the peer ⏳ If the timer expires during synchronization with the peer, any stale routes the peer hasn't refreshed are removed 🙈 And if the session resets again before becoming synchronized, any remaining routes (for the AFI/SAFI whose LLST timer expired) MUST be removed immediately 😭🔥

4.3. Processing LLGR_STALE Routes ✨

Okay bestie, so if a BGP speaker has advertised the Long-Lived Graceful Restart Capability to a neighbor, it MUST do the following 💅 whenever it gets a route from that neighbor carrying the LLGR_STALE community, or when it attaches LLGR_STALE itself per Section 4.2:

• Treat the route as the least preferred in route selection (see below) 🙈 See Section 5.2 for a discussion of potential risks inherent in doing this.
• The route SHOULD NOT be advertised to any neighbor from which the Long-Lived Graceful Restart Capability has not been received. The exception is described in Section 4.6. Note that this requirement implies that such routes should be withdrawn from any such neighbor. 🫶
• The LLGR_STALE community MUST NOT be removed when the route is further advertised. Do not touch it, it's giving protected 🔒

4.4. Route Selection

A least preferred route MUST be treated as less preferred than any other route that is not also least preferred. The rule is literally "least preferred loses to everyone else who isn't also least preferred" 👑

When performing route selection between two routes when both are least preferred, normal tiebreaking applies. Note that this would only be expected to happen if the only routes available for selection were least preferred; in all other cases, such routes would have been eliminated from consideration. So it's basically a last-resort situation, iykyk 💀

4.5. Errors

If the LLGR Capability is received without an accompanying GR Capability, the LLGR Capability MUST be ignored, that is, the implementation MUST behave as though no LLGR Capability has been received. No GR Capability, no LLGR. That's the vibe 🚫✨

4.6. Optional Partial Deployment Procedure

Ideally, all routers in an Autonomous System (AS) would support this specification before it were enabled. 🌈 However, to facilitate incremental deployment, stale routes MAY be advertised to neighbors that have not advertised the Long-Lived Graceful Restart Capability under the following conditions:

• The neighbors MUST be internal (Internal BGP (IBGP) or Confederation) neighbors.
• The NO_EXPORT community [RFC1997] MUST be attached to the stale routes.
• The stale routes MUST have their LOCAL_PREF set to zero. See Section 5.2 for a discussion of potential risks inherent in doing this. ⚠️

If this strategy for partial deployment is used, the network operator should set the LOCAL_PREF to zero for all long-lived stale routes throughout the Autonomous System. This trades off a small reduction in flexibility (ordering may not be preserved between competing long-lived stale routes) for consistency between routers that do, and do not, support this specification. Since the consistency of route selection can be important for preventing forwarding loops, the latter consideration dominates. Consistency is the main character here 🎀

4.7. Procedures When BGP Is the PE-CE Protocol in a VPN

4.7.1. Procedures When EBGP Is the PE-CE Protocol in a VPN

In VPN deployments (for example, [RFC4364]), External BGP (EBGP) is often used as a PE-CE protocol. It may be a practical necessity in such deployments to accommodate interoperation with peer routers that cannot easily be upgraded to support specifications such as this one. This leads to a problem: the procedures defined elsewhere in this document generally prevent LLGR stale routes from being sent across EBGP sessions that don't support LLGR, but this could prevent the VPN routes from being used for their intended purpose. Ugh, the classic "the rules are too strict for real life" plot twist 😭

We observe that the principal motivation for restricting the propagation of "stale" routing information is the desire to prevent it from spreading without limit once it exits the "safe" perimeter. We further observe that VPN deployments are typically topologically constrained, making this concern moot. For this reason, an implementation MAY advertise stale routes over a PE-CE session, when explicitly configured to do so. That is, the second rule listed in Section 4.3 MAY be disregarded in such cases. All other rules continue to apply. Finally, if this exception is used, the implementation SHOULD, by default, attach the NO_EXPORT community to the routes in question, as an additional protection against stale routes spreading without limit. Attachment of the NO_EXPORT community MAY be disabled by explicit configuration in order to accommodate exceptional cases. Exceptions to the exception, very dramatic, very on-brand 🧚

See further discussion of using an explicitly configured policy to mitigate this issue in Section 5.1.

4.7.2. Procedures When IBGP Is the PE-CE Protocol in a VPN

If IBGP is used as the PE-CE protocol, following the procedures of [RFC6368], then when a PE router imports a VPN route that contains the ATTR_SET attribute into a destination VRF and subsequently advertises that route to a CE router:

• If the CE router supports the procedures of this document (in other words, if the CE router has advertised the LLGR Capability):
  • In addition to including the path attributes derived from the ATTR_SET attribute in the advertised route as per [RFC6368], the PE router MUST also include the LLGR_STALE community if it is present in the path attributes of the imported route, even if it is not present in the ATTR_SET attribute. 🦋
• If the CE router does not support the procedures of this document:
  • Then the optional procedures of Section 4.6 MAY be followed, attaching the NO_EXPORT community and setting the value of LOCAL_PREF to zero, overriding the value found in the ATTR_SET.

Similarly, when a PE router receives a route from a CE into its VRF and subsequently exports that route to a VPN address family:

• If the PE router supports the procedures of this document (in other words, if the PE router has advertised the LLGR Capability):
  • In addition to including in the VPN route the ATTR_SET derived from the path attributes as per [RFC6368], the PE router MUST also include the LLGR_STALE community in the VPN route if it is present in the path attributes of the route as received from the CE. 💖
• If the PE router does not support the procedures of this document:
  • There exists no ideal solution. The CE could advertise a route with LLGR_STALE, with the understanding that the LLGR_STALE marking will only be honored by the provider network if appropriate policy configuration exists on the PE (see Section 5.1). It is at least guaranteed that LLGR_STALE will be propagated when the route is propagated beyond the provider network, or the CE could refrain from advertising the LLGR_STALE route to the incapable PE. No perfect answer here, sadly 😔

---

5. Deployment Considerations 🌸

The deployment considerations discussed in [RFC4724] apply to this document. In addition, network operators are cautioned to carefully consider the potential disadvantages of deploying these procedures for a given AFI/SAFI. Most notably, if used for an AFI/SAFI that conveys conventional reachability information, the use of a long-lived stale route could result in a loss of connectivity for the covered prefix. This specification takes pains to mitigate this risk where possible by making such routes least preferred and by restricting the scope of such routes to routers that support these procedures (or, optionally, a single Autonomous System, see Section 4.6). However, if a stale route is chosen as best for a given prefix, then according to the normal rules of IP forwarding, that route will be used for matching destinations, even if a non-stale less specific matching route is also available. Networks in which the deployment of these procedures would be especially concerning include those that do not use "tunneled" forwarding (in other words, those using conventional hop-by-hop forwarding). Real talk: please be careful with this one, it's serious 🚨

Implementations MUST NOT enable these procedures by default. They MUST require affirmative configuration per AFI/SAFI in order to enable them. Opt-in only, no surprise parties 🎉

The procedures of this document do not alter the route resolvability requirement of Section 9.1.2.1 of [RFC4271]. Because of this, it will commonly be the case that "stale" IBGP routes will only continue to be used if the router depicted in the next hop remains resolvable, even if its BGP component is down. Details of IGP fault-tolerance strategies are beyond the scope of this document. In addition to the foregoing, it may be advisable to check the viability of the next hop through other means, for example, Bidirectional Forwarding Detection (BFD) [RFC5880]. This may be especially useful in cases where the next hop is known directly at the network layer, notably EBGP. ✨

As discussed in this document, after a BGP session goes down and before the session is re-established, stale routes may be retained for up to two consecutive periods, controlled by the Restart Time and the Long-Lived Stale Time, respectively:

• During the first period, routing churn would be prevented, but with potential persistent packet loss.
• During the second period, potential persistent packet loss may be reduced, but routing churn would be visible throughout the network.

The setting of the relevant parameters for a particular application should take into account trade-offs, network dynamics, and potential failure scenarios. If needed, the first period can be bypassed either by local configuration or by setting the Restart Time in the Graceful Restart Capability to zero and/or not listing the AFI/SAFI in that capability. 🍓

The setting of the F bit (and the Forwarding State bit of the accompanying GR Capability) depends, in part, on deployment considerations. The F bit can be understood as an indication that the Helper should flush associated routes (if the bit is left clear). As discussed in Section 1, an important use case for LLGR is for routes that are more akin to configuration than to conventional routing. For such routes, it may make sense to always set the F bit, regardless of other considerations. Likewise, for control-plane-only entities, such as dedicated route reflectors that do not participate in the forwarding plane, it makes sense to always set the F bit. Overall, the rule of thumb is that if loss of state on the restarting router can reasonably be expected to cause a forwarding loop or persistent packet loss, the F bit should be set scrupulously according to whether state has been retained. Specifics of whether or not the F bit is set are implementation dependent and may also be controlled by configuration. Also, for every AFI/SAFI represented in the LLGR Capability that is also represented in the GR Capability, there will be two corresponding F bits: the LLGR F bit and the GR F bit. If the LLGR F bit is set, the corresponding GR F bit should also be set, since to do otherwise would cause the state to be cleared on the Receiving Router per the normal rules of GR, violating the intent of the set LLGR bit. Two F bits, one vibe, don't mess it up 🦋💜

5.1. When BGP Is the PE-CE Protocol in a VPN 🎀

okay so, remember Section 4.7? It says a PE might need to advertise stale routes to a CE in some VPN setups, even if that CE doesn't support this spec 🙈 When that happens, the operator setting up the PE needs to tell the operator on the CE side, and the CE should be configured to depreference those routes. 💅

It works the other way too! A CE might need to advertise stale routes to a PE that doesn't support this spec. Same deal: the operator setting up the CE tells the operator on the PE side, and the PE should be configured to depreference the routes. 🫶

Good news: typical BGP implementations can usually be set up to depreference routes by matching on the LLGR_STALE community and setting the LOCAL_PREF for matching routes to zero. It's basically the same move as the one in Section 4.6, just aimed at a different spot ✨

5.2. Risks of Depreferencing Routes 💀

Depreferencing EBGP routes is considered safe, no different from the super common practice of putting a routing policy on an EBGP session. Lowkey easy. 🌸

IBGP is a whole different story though 😭

Consistent route selection is a fundamental tenet of IBGP correctness and safe operation in hop-by-hop routed networks. Like, it's the whole foundation. If routers inside an AS apply different criteria when they pick routes, they can end up with inconsistent route selections. That can cause forwarding loops, unless some kind of tunneled forwarding is used to stop "core" routers from making a (possibly inconsistent) forwarding decision based on the IP header. Yikes. 👑

This spec uses the state of a peering session as an input to the selection criteria. It depreferences routes tied to a session that has gone down but hasn't aged out yet. Here's the tea: different routers in the same AS might disagree about whether their sessions with a given peer are up or down. So they could apply different selection criteria to routes from that peer, and then a forwarding loop could form between them. 🔥

Let's look at an example of that loop with a simple topology:

   A ---- B ---- C ------------------------- D
   ^                                         ^
   |                                         |
   R1                                        R2

                                  Figure 1

In this example, A - D are routers with a full mesh of IBGP sessions between them (the sessions aren't shown). The short links have unit cost, and the long link has cost 5. Routers A and D are AS border routers, and each one advertises some route, R, with the same LOCAL_PREF into the AS. Those are R1 and R2 in the diagram. In ordinary operation, B and C both select R1 for forwarding and forward toward A. ✨

Now suppose the session between A and B goes down for some reason, and it stays down long enough for LLGR processing to kick in on B. Then, on B, route R1 gets depreferenced, so B selects R2 instead. But C keeps preferring R1. Bestie, that's the problem: a forwarding loop for packets destined to R forms between B and C. 😭🌀 (Fyi, other forwarding loop scenarios can be built for conventional GR too, but those are generally considered less severe, since GR can only stay in effect for a much shorter interval.)

The potential benefits of this spec can outweigh the risks above, as long as you're careful when deploying it. The cardinal rule is this: if a given set of routes is being used within an AS for hop-by-hop forwarding, enabling LLGR procedures is not recommended. 🚫 If tunneled forwarding (such as MPLS) is used within the AS, or if routes are being used for purposes other than hop-by-hop forwarding, less caution is needed. The operator should still carefully think through the consequences of enabling LLGR, though. Don't skip that part, periodt 💅

---

6. Security Considerations 🔐

The security implications of the LLGR mechanism in this document are akin to those that come with keeping stale routing info around in a network. But here's the catch: the retention time can be way longer, so the window for certain attacks to work could grow a lot. 😱 This matters most when you're keeping routing info that's used for service segregation, like MPLS label entries.

For MPLS VPN services, traffic isolation between VPNs depends on the MPLS labels being correct between ingress and egress PEs. Specifically, when an egress PE withdraws a label L1 that was allocated to a VPN1 route, that label must not be assigned to a VPN route from a different VPN until all ingress PEs have stopped using the old VPN1 route that uses L1. Not me getting paranoid, but this is a big deal 🙈

This kind of corner case can already happen today, if propagating VPN routes between PEs via BGP messages takes longer than the label reallocation delay on a PE. We can generally bound the worst-case BGP propagation time to a few minutes (for example, 2-5 minutes). So the security breach won't happen if PEs are designed to not reallocate a previously used and withdrawn label before a few minutes have passed. 🧚

It gets worse with BGP GR between PEs, because VPN routes can be stalled for a longer period of time (for example, 20 minutes). 😬

Then LLGR makes it even more intense, since VPN routes can be stalled for a much longer period of time (for example, 2 hours, 1 day). Yikes 💀

To actually exploit the vulnerability above, an attacker would need to engineer a specific LLGR state between two PE devices, and also make the label reallocation happen in a way that the two topologies overlap. To avoid a potential VPN breach, the operator should make sure the lower bound for label reuse is greater than the upper bound on the LLST before enabling LLGR for a VPN address family. Section 4.2 talks about providing an upper bound on LLST. Features for setting a lower bound on label reuse time are beyond the scope of this document, but factors that might need to be taken into account when setting that value include:

• The load of the BGP route churn on a PE (in terms of the number of VPN labels advertised and the churn rate).
• The label allocation policy on the PE, which possibly depends on the size of the pool of VPN labels (which can be restricted by hardware considerations or other MPLS usages), the label allocation scheme (for example, per route or per VRF/CE), and the reallocation policy (for example, least recently used label).

Note that [RFC4781], which defines the Graceful Restart Mechanism for BGP with MPLS, is also applicable to LLGR. 🌈

7. Examples of Operation ✨

okay so let's walk through a few examples of how this could actually play out 💖 these are just for illustration. They're not exhaustive and they're not normative, so the real rules are everything above this section 🫶

Picture this: there's a border router called ASBR1 with an IBGP peering to a route reflector called RR1, and that's where it learns its routes from 🛣️ It also has an EBGP peering with an external peer called EXT, and it advertises those routes over to EXT. EXT has advertised the GR and LLGR Capabilities to ASBR1, and ASBR1 is set up to do GR and LLGR on its sessions with both RR1 and EXT. RR1 advertises a GR Restart Time of 1 (second) and an LLST of 3600 (seconds) ⏱️

    +==========+=====================================================+
    | Time     | Event                                               |
    +==========+=====================================================+
    | t        | ASBR1's IBGP session with RR fails.  ASBR1 retains  |
    |          | RR's routes according to the rules of GR [RFC4724]. |
    +----------+-----------------------------------------------------+
    | t+1      | GR Restart Time expires.  ASBR1 transitions RR's    |
    |          | routes to long-lived stale routes by attaching the  |
    |          | LLGR_STALE community and depreferencing them.       |
    |          | However, since it has no backup routes, it          |
    |          | continues to make use of them.  It re-announces     |
    |          | them to EXT with the LLGR_STALE community attached. |
    +----------+-----------------------------------------------------+
    | t+1+3600 | LLST expires.  ASBR1 removes RR's stale routes from |
    |          | its own RIB and sends BGP updates to withdraw them  |
    |          | from EXT.                                           |
    +----------+-----------------------------------------------------+

                                 Table 1

Next, same scenario, but this time RR1 advertised a GR Restart Time of zero, which effectively turns GR off 🙈 Or, ASBR1 could've used a local config to override the Restart Time RR1 offered and set it to a local value of zero instead. Either way, the result looks like this 👇

   +==========+=======================================================+
   | Time     | Event                                                 |
   +==========+=======================================================+
   | t        | ASBR1's IBGP session with RR fails.  ASBR1            |
   |          | transitions RR's routes to long-lived stale routes by |
   |          | attaching the LLGR_STALE community and depreferencing |
   |          | them.  However, since it has no backup routes, it     |
   |          | continues to make use of them.  It re-announces them  |
   |          | to EXT with the LLGR_STALE community attached.        |
   +----------+-------------------------------------------------------+
   | t+0+3600 | LLST expires.  ASBR1 removes RR's stale routes from   |
   |          | its own RIB and sends BGP updates to withdraw them    |
   |          | from EXT.                                             |
   +----------+-------------------------------------------------------+

                                 Table 2

Next, back to the original scenario, but now the ASBR1-RR1 session comes back up and gets resynchronized 180 seconds after the failure was detected. Honestly this is the comeback arc we love to see 🥹✨

     +=========+=====================================================+
     | Time    | Event                                               |
     +=========+=====================================================+
     | t       | ASBR1's IBGP session with RR fails.  ASBR1 retains  |
     |         | RR's routes according to the rules of GR [RFC4724]. |
     +---------+-----------------------------------------------------+
     | t+1     | GR Restart Time expires.  ASBR1 transitions RR's    |
     |         | routes to long-lived stale routes by attaching the  |
     |         | LLGR_STALE community and depreferencing them.       |
     |         | However, since it has no backup routes, it          |
     |         | continues to make use of them.  It re-announces     |
     |         | them to EXT with the LLGR_STALE community attached. |
     +---------+-----------------------------------------------------+
     | t+1+179 | Session is re-established and resynchronized.       |
     |         | ASBR1 removes the LLGR_STALE community from RR1's   |
     |         | routes and re-announces them to EXT with the        |
     |         | LLGR_STALE community removed.                       |
     +---------+-----------------------------------------------------+

                                  Table 3

Last one! Original scenario again, but this time EXT has not advertised the LLGR Capability to ASBR1. Plot twist: ASBR1 can't just re-announce the stale routes, so it withdraws them from EXT instead 😭

    +==========+======================================================+
    | Time     | Event                                                |
    +==========+======================================================+
    | t        | ASBR1's IBGP session with RR fails.  ASBR1 retains   |
    |          | RR's routes according to the rules of GR [RFC4724].  |
    +----------+------------------------------------------------------+
    | t+1      | GR Restart Time expires.  ASBR1 transitions RR's     |
    |          | routes to long-lived stale routes by attaching the   |
    |          | LLGR_STALE community and depreferencing them.        |
    |          | However, since it has no backup routes, it continues |
    |          | to make use of them.  It withdraws them from EXT.    |
    +----------+------------------------------------------------------+
    | t+1+3600 | LLST expires.  ASBR1 removes RR's stale routes from  |
    |          | its own RIB.                                         |
    +----------+------------------------------------------------------+

                                  Table 4

---

8. IANA Considerations 🎀

okay IANA stuff, but make it cute 🦋 This document defines a BGP capability called the "Long-Lived Graceful Restart Capability", and IANA gave it the value 71 from the "Capability Codes" registry 💜

It also introduces two BGP well-known communities:

• the first is called LLGR_STALE, and it's for marking long-lived stale routes, and
• the second is called NO_LLGR, and it's for marking routes that should not be retained if stale.

IANA assigned these well-known community values 0xFFFF0006 and 0xFFFF0007, respectively, from the "BGP Well-known Communities" registry 🌸

IANA has also set up a registry called the "Long-Lived Graceful Restart Flags for Address Family" registry, under the "Border Gateway Protocol (BGP) Parameters" group ✨ The registration procedures are Standards Action (see [RFC8126]). It's initially populated like this:

     +==============+=======================+============+===========+
     | Bit Position | Name                  | Short Name | Reference |
     +==============+=======================+============+===========+
     | 0            | Preservation of state | F          | RFC 9494  |
     +--------------+-----------------------+------------+-----------+
     | 1-7          | Unassigned            |            |           |
     +--------------+-----------------------+------------+-----------+

                                  Table 5

---

9. References 📚

9.1. Normative References

   [RFC1997]  Chandra, R., Traina, P., and T. Li, "BGP Communities
              Attribute", RFC 1997, DOI 10.17487/RFC1997, August 1996,
              <https://www.rfc-editor.org/info/rfc1997>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
              Border Gateway Protocol 4 (BGP-4)", RFC 4271,
              DOI 10.17487/RFC4271, January 2006,
              <https://www.rfc-editor.org/info/rfc4271>.

   [RFC4724]  Sangli, S., Chen, E., Fernando, R., Scudder, J., and Y.
              Rekhter, "Graceful Restart Mechanism for BGP", RFC 4724,
              DOI 10.17487/RFC4724, January 2007,
              <https://www.rfc-editor.org/info/rfc4724>.

   [RFC4760]  Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
              "Multiprotocol Extensions for BGP-4", RFC 4760,
              DOI 10.17487/RFC4760, January 2007,
              <https://www.rfc-editor.org/info/rfc4760>.

   [RFC5492]  Scudder, J. and R. Chandra, "Capabilities Advertisement
              with BGP-4", RFC 5492, DOI 10.17487/RFC5492, February
              2009, <https://www.rfc-editor.org/info/rfc5492>.

   [RFC6368]  Marques, P., Raszuk, R., Patel, K., Kumaki, K., and T.
              Yamagata, "Internal BGP as the Provider/Customer Edge
              Protocol for BGP/MPLS IP Virtual Private Networks (VPNs)",
              RFC 6368, DOI 10.17487/RFC6368, September 2011,
              <https://www.rfc-editor.org/info/rfc6368>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8538]  Patel, K., Fernando, R., Scudder, J., and J. Haas,
              "Notification Message Support for BGP Graceful Restart",
              RFC 8538, DOI 10.17487/RFC8538, March 2019,
              <https://www.rfc-editor.org/info/rfc8538>.

9.2. Informative References

   [RFC4364]  Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
              Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
              2006, <https://www.rfc-editor.org/info/rfc4364>.

   [RFC4761]  Kompella, K., Ed. and Y. Rekhter, Ed., "Virtual Private
              LAN Service (VPLS) Using BGP for Auto-Discovery and
              Signaling", RFC 4761, DOI 10.17487/RFC4761, January 2007,
              <https://www.rfc-editor.org/info/rfc4761>.

   [RFC4781]  Rekhter, Y. and R. Aggarwal, "Graceful Restart Mechanism
              for BGP with MPLS", RFC 4781, DOI 10.17487/RFC4781,
              January 2007, <https://www.rfc-editor.org/info/rfc4781>.

   [RFC5880]  Katz, D. and D. Ward, "Bidirectional Forwarding Detection
              (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010,
              <https://www.rfc-editor.org/info/rfc5880>.

   [RFC8126]  Cotton, M., Leiba, B., and T. Narten, "Guidelines for
              Writing an IANA Considerations Section in RFCs", BCP 26,
              RFC 8126, DOI 10.17487/RFC8126, June 2017,
              <https://www.rfc-editor.org/info/rfc8126>.

   [RFC8955]  Loibl, C., Hares, S., Raszuk, R., McPherson, D., and M.
              Bacher, "Dissemination of Flow Specification Rules",
              RFC 8955, DOI 10.17487/RFC8955, December 2020,
              <https://www.rfc-editor.org/info/rfc8955>.

---

Acknowledgements 🫶

We'd like to thank Nabil Bitar, Martin Djernaes, Roberto Fragassi, Jeffrey Haas, Jakob Heitz, Daniam Henriques, Nicolai Leymann, Mike McBride, Paul Mattes, John Medamana, Pranav Mehta, Han Nguyen, Saikat Ray, Valery Smyslov, and Bo Wu for their valuable input and contributions to the discussion and solution 💕

Contributors 💜

   Clarence Filsfils
   Cisco Systems
   1150 Brussels
   Belgium
   Email: cf@cisco.com


   Pradosh Mohapatra
   Sproute Networks
   Email: mpradosh@yahoo.com


   Yakov Rekhter


   Eric Rosen
   Email: erosen52@gmail.com


   Rob Shakir
   Google, Inc.
   1600 Amphitheatre Parkway
   Mountain View, CA 94043
   United States of America
   Email: robjs@google.com


   Adam Simpson
   Nokia
   Email: adam.1.simpson@nokia.com

Authors' Addresses 💖

okay bestie, here are the four brilliant humans who wrote this RFC and how to reach each of them 🫶

James Uttaro 🎀 Independent Contributor Email: juttaro@ieee.org

Enke Chen 🦋 Palo Alto Networks Email: enchen@paloaltonetworks.com

Bruno Decraene 🌸 Orange Email: bruno.decraene@orange.com

John G. Scudder 👑 Juniper Networks Email: jgs@juniper.net

that's the whole squad, and they are iconic for this one 💅✨