RFC 8326

Graceful BGP Session Shutdown

📅 published March 2018, from ietf

Graceful BGP Session Shutdown 💖

okay bestie, let's get into it! 🎀 This one is an IETF Standards Track doc, written by P. Francois, Ed. (Individual Contributor), B. Decraene, Ed. (Orange), C. Pelsser (Strasbourg University), K. Patel (Arrcus, Inc.), and C. Filsfils (Cisco Systems). It's RFC 8326, published in March 2018, and it's on the ISSN 2070-1721 stream. It doesn't obsolete or update anything, so it's a totally fresh start 🌸✨

The whole point is a new well-known BGP community called GRACEFUL_SHUTDOWN 🦋, plus some operator procedures that use it to cut down on traffic getting dropped when a BGP peering session gets shut down on purpose, like for planned maintenance 🔧💅

Status of This Memo 📜

This is an Internet Standards Track document. 💜

It's a product of the Internet Engineering Task Force (IETF) 🌈 and represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). For more on Internet Standards, check Section 2 of RFC 7841 🫶

Info on the current status, any errata, and how to give feedback lives at https://www.rfc-editor.org/info/rfc8326 ✨

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. 🎀

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document. Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License. 🙈

Table of Contents 📚

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   4
3.  Packet Loss upon Manual EBGP Session Shutdown . . . . . . . .   4
4.  Procedure for EBGP Graceful Shutdown  . . . . . . . . . . . .   4
  4.1.  Pre-configuration . . . . . . . . . . . . . . . . . . . .   5
  4.2.  Operations at Maintenance Time  . . . . . . . . . . . . .   5
  4.3.  BGP Implementation Support for Graceful Shutdown  . . . .   6
5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   6
6.  Security Considerations . . . . . . . . . . . . . . . . . . .   6
7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   6
  7.1.  Normative References  . . . . . . . . . . . . . . . . . .   6
  7.2.  Informative References  . . . . . . . . . . . . . . . . .   7
Appendix A.  Alternative Techniques with Limited Applicability  .   8
  A.1.  Multi-Exit Discriminator Tweaking . . . . . . . . . . . .   8
  A.2.  IGP Distance Poisoning  . . . . . . . . . . . . . . . . .   8
Appendix B.  Configuration Examples . . . . . . . . . . . . . . .   8
  B.1.  Cisco IOS XR  . . . . . . . . . . . . . . . . . . . . . .   9
  B.2.  BIRD  . . . . . . . . . . . . . . . . . . . . . . . . . .   9
  B.3.  OpenBGPD  . . . . . . . . . . . . . . . . . . . . . . . .  10
Appendix C.  Beyond EBGP Graceful Shutdown  . . . . . . . . . . .  10
  C.1.  IBGP Graceful Shutdown  . . . . . . . . . . . . . . . . .  10
  C.2.  EBGP Session Establishment  . . . . . . . . . . . . . . .  10
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . .  12
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  12

1. Introduction 🌷

Routing changes in BGP can come from planned maintenance 🛠️ This doc defines a well-known community [RFC1997], called GRACEFUL_SHUTDOWN, to make gracefully shutting down BGP sessions way less of a management headache 😭 The community lets implementers build an automated graceful shutdown mechanism that doesn't need any router reconfiguration at maintenance time. Iconic ✨

The doc walks through operational procedures for cutting down or wiping out packet loss during a maintenance operation 💅 The loss comes from the brief lack of reachability during BGP convergence that follows shutting down an EBGP peering session between two Autonomous System Border Routers (ASBRs) 🦋

This doc covers cases where maintenance affects the forwarding plane, so it's for when Graceful Restart doesn't apply 🧚

The procedures can cut down or avoid packet loss for outbound and inbound traffic that was first forwarded along the peering link being shut down. In both Autonomous Systems (ASes), these procedures trigger rerouting to alternate paths if they exist within the AS, while still letting the old path get used until alternate ones are learned. That means routers always have a valid route during convergence, which is so main character energy 👑

The goal is to meet the requirements in [RFC6198] as best possible without changing BGP 🌈

Other maintenance cases, like shutting down an IBGP session or establishing an EBGP session, are out of scope here. For info, they get a quick look in Appendix C 🫶

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 🎀⚠️

Heads up: those key words only count as requirements when they're in ALL CAPS, like the ones above. 💀

2. Terminology 📖

graceful shutdown initiator A router on which the session shutdown is performed for the maintenance. 🔧

graceful shutdown receiver A router that has a BGP session to be shut down with the graceful shutdown initiator. 🧚

3. Packet Loss upon Manual EBGP Session Shutdown 💔

Packets can get lost during BGP convergence after someone manually shuts down an EBGP session, and there are two reasons why 😩

First, some routers can end up with no path toward an affected prefix, so they drop traffic headed there. That's because alternate paths can get hidden by nodes inside an AS. This happens when the extension defined in [RFC7911] isn't used and either a) the paths aren't selected as best by the ASBRs that receive them over an EBGP session, or b) Route Reflectors don't pass the paths further along the IBGP topology because they didn't select them as best 🙈

Second, the FIB can be inconsistent between routers within the AS. Packets toward affected prefixes can then loop and get dropped, unless encapsulation is used within the AS 🌀

This document only tackles the first reason 💅

4. Procedure for EBGP Graceful Shutdown ✨

This section describes the configurations and actions to perform for gracefully shutting down EBGP peering links 🦋

The goal is to keep the paths being shut down between the peers, but with a lower LOCAL_PREF value 🫶 That way the paths stay in use while alternate paths get selected and propagated, instead of just withdrawing the paths outright. The LOCAL_PREF value SHOULD be lower than any of the alternative paths. The RECOMMENDED value is 0. 🌸

Some alternative techniques with limited applicability are covered in Appendix A, just for info 🧚

4.1. Pre-configuration 🛠️

On each ASBR that supports the graceful shutdown receiver procedure, an inbound BGP route policy gets applied on all of the ASBR's EBGP sessions. That policy:

  • matches the GRACEFUL_SHUTDOWN community.
  • sets the LOCAL_PREF attribute of the paths tagged with the GRACEFUL_SHUTDOWN community to a low value.

Examples of configurations are in Appendix B, for info 💜

4.2. Operations at Maintenance Time ⏰

On the graceful shutdown initiator, at maintenance time, the operator:

  • applies an outbound BGP route policy on the EBGP session to be shut down. This policy tags the paths propagated over the session with the GRACEFUL_SHUTDOWN community. This will trigger the BGP implementation to re-advertise all active routes previously advertised and tag them with the GRACEFUL_SHUTDOWN community.
  • applies an inbound BGP route policy on the EBGP session to be shut down. This policy tags the paths received over the session with the GRACEFUL_SHUTDOWN community and sets LOCAL_PREF to a low value.
  • waits for route re-advertisement over the EBGP session and for BGP routing convergence on both ASBRs.
  • shuts down the EBGP session, optionally using [RFC8203] to communicate the reason for the shutdown.

If the whole router is shutting down, then on top of gracefully shutting down all the EBGP sessions, you also need to gracefully shut down the routes this router originates (e.g., BGP aggregates redistributed from other protocols, including static routes). You do that by tagging these routes with the GRACEFUL_SHUTDOWN community and setting LOCAL_PREF to a low value 🎀

4.3. BGP Implementation Support for Graceful Shutdown 🌈

BGP Implementers SHOULD provide configuration knobs that use the GRACEFUL_SHUTDOWN community to inform BGP neighbors ahead of an impending neighbor shutdown. Implementation details are outside the scope of this document 💖

5. IANA Considerations 🗂️

IANA previously assigned the community value 0xFFFF0000 to the 'planned-shut' community in the "BGP Well-known Communities" registry. IANA has changed the name 'planned-shut' to 'GRACEFUL_SHUTDOWN' and updated the reference to point to this document 🦋

6. Security Considerations 🔐

By offering the graceful shutdown service to a neighboring AS, an ISP gives that neighbor, and possibly its downstream ASes, a way to lower the LOCAL_PREF value assigned to paths received from this neighbor. 😳

The neighbor could abuse this and do inbound traffic engineering by declaring that some prefixes are under maintenance, so they can shift traffic to another peering link. 💀

If the ISP doesn't tolerate this behavior, it SHOULD monitor the use of the graceful shutdown community 🚨

7. References 📚

Part 2 picks up the reference lists, the appendices, and the rest, so stay tuned, bestie 🫶💖

7.1. Normative References

okay bestie, these are the must-read refs 📚✨ the ones you NEED for the spec to make sense. copied exactly so nothing gets lost:

[RFC1997]  Chandra, R., Traina, P., and T. Li, "BGP Communities
           Attribute", RFC 1997, DOI 10.17487/RFC1997, August 1996,
           <https://www.rfc-editor.org/info/rfc1997>.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC6198]  Decraene, B., Francois, P., Pelsser, C., Ahmad, Z.,
           Elizondo Armengol, A., and T. Takeda, "Requirements for
           the Graceful Shutdown of BGP Sessions", RFC 6198,
           DOI 10.17487/RFC6198, April 2011,
           <https://www.rfc-editor.org/info/rfc6198>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

7.2. Informative References

these ones are extra reading 🦋 nice to have, not strictly needed:

[BEST-EXTERNAL]
           Marques, P., Fernando, R., Chen, E., Mohapatra, P., and H.
           Gredler, "Advertisement of the best external route in
           BGP", Work in Progress, draft-ietf-idr-best-external-05,
           January 2012.

[RFC7911]  Walton, D., Retana, A., Chen, E., and J. Scudder,
           "Advertisement of Multiple Paths in BGP", RFC 7911,
           DOI 10.17487/RFC7911, July 2016,
           <https://www.rfc-editor.org/info/rfc7911>.

[RFC8203]  Snijders, J., Heitz, J., and J. Scudder, "BGP
           Administrative Shutdown Communication", RFC 8203,
           DOI 10.17487/RFC8203, July 2017,
           <https://www.rfc-editor.org/info/rfc8203>.

Appendix A. Alternative Techniques with Limited Applicability 🌸

okay so a few other techniques were looked at for graceful shutdown, but they got rejected because they only work in limited situations 🙈 this appendix describes them for reference, so you know what was considered and why it didn't make the cut.

A.1. Multi-Exit Discriminator Tweaking

the idea here is to bump up the Multi-Exit Discriminator (MED) on the paths you want to avoid, so the routers in the neighboring AS pick other paths instead 💅

but here's the catch: it only works if the alternate paths are just as good as the original ones when it comes to LOCAL_PREF and AS Path Length. if they aren't, raising the MED won't change what the neighboring AS's routers decide. not it 💀

A.2. IGP Distance Poisoning

this one makes the distance to the BGP NEXT_HOP for the maintained session bigger inside the IGP, so the old paths lose out during the IGP distance tie-break rule ✨

again, it only works for paths whose alternates are as good as the old ones on LOCAL_PREF, AS Path length, and MED. the vibe is still "not quite" 😭

and also, this poisoning can't be applied when BGP NEXT_HOP self is used, because there's no BGP NEXT_HOP specific to the maintained session to poison in the IGP. no target, no poison 🙃

Appendix B. Configuration Examples 🎀

This appendix is non-normative.

ok, here's where we get to see real config! 🍓 this appendix has examples of routing policy configs that honor the GRACEFUL_SHUTDOWN well-known BGP community.

B.1. Cisco IOS XR

community-set comm-graceful-shutdown
  65535:0
end-set
!
route-policy AS64497-ebgp-inbound
  ! normally this policy would contain much more
  if community matches-any comm-graceful-shutdown then
    set local-preference 0
  endif
end-policy
!
router bgp 64496
 neighbor 2001:db8:1:2::1
  remote-as 64497
  address-family ipv6 unicast
   send-community-ebgp
   route-policy AS64497-ebgp-inbound in

  !
 !
!

B.2. BIRD

function honor_graceful_shutdown() {
    if (65535, 0) ~ bgp_community then {
        bgp_local_pref = 0;
    }
}
filter AS64497_ebgp_inbound
{
        # normally this policy would contain much more
        honor_graceful_shutdown();
}
protocol bgp peer_64497_1 {
    neighbor 2001:db8:1:2::1 as 64497;
    local as 64496;
    import keep filtered;
    import filter AS64497_ebgp_inbound;
}

B.3. OpenBGPD

AS 64496
router-id 192.0.2.1
neighbor 2001:db8:1:2::1 {
        remote-as 64497
}
# normally this policy would contain much more
match from any community GRACEFUL_SHUTDOWN set { localpref 0 }

Appendix C. Beyond EBGP Graceful Shutdown 🌈

this last appendix is about the stuff that goes beyond EBGP graceful shutdown, like IBGP and session establishment. it's giving bonus content 🧚

C.1. IBGP Graceful Shutdown

for shutting down an IBGP session, as long as the IBGP topology stays viable after maintenance (so all BGP speakers of the AS have an IBGP signaling path for all prefixes advertised on this graceful shutdown IBGP session), shutting it down does not lead to transient unreachability 🙌 which means no specific graceful shutdown action is required.

C.2. EBGP Session Establishment

there are two potential causes for transient packet losses when an EBGP session comes up. the first one is local to the startup initiator. the second one comes from BGP convergence after new best paths get injected into the IBGP topology 👀

C.2.1. Unreachability Local to the ASBR

an ASBR that picks a path received over a newly established EBGP session as its best path may transiently drop traffic. this can typically happen when the NEXT_HOP attribute differs from the IP address of the EBGP peer, and the receiving ASBR hasn't resolved yet the MAC address associated with the IP address of that third-party NEXT_HOP 😭

a BGP speaker implementation MAY avoid such losses by making sure third-party NEXT_HOPs are resolved before installing paths that use these NEXT_HOPs in the RIB ✨

or, the operator (script) MAY ping third-party NEXT_HOPs that are expected to be used before establishing the session. doing this means the MAC addresses for those third-party NEXT_HOPs get resolved by the startup initiator 🫶

C.2.2. IBGP Convergence

during EBGP session establishment, in some corner cases, a router may have no path toward an affected prefix, which leads to loss of connectivity. yikes 💀

here's a typical example of that transient unreachability for a given prefix. picture three Route Reflectors (RR): RR1, RR2, and RR3, with a full mesh of IBGP sessions between them 🕸️

  1. RR1 is initially advertising the current best path to the members of its IBGP RR full mesh. it propagated that path within its RR full mesh. RR2 knows only that path toward the prefix.
  2. RR3 receives a new best path originated by the startup initiator, which is one of its RR clients. RR3 selects it as best and propagates an UPDATE within its RR full mesh, i.e., to RR1 and RR2.
  3. RR1 receives that path, reruns its decision process, and picks this new path as best. as a result, RR1 withdraws its previously announced best path on the IBGP sessions of its RR full mesh.
  4. if, for any reason, RR3 processes the withdraw generated in step 3 before processing the update generated in step 2, RR3 transiently suffers from unreachability for the affected prefix. oof 😩

the fix-it squad here is [RFC7911] or [BEST-EXTERNAL] among the RRs of the IBGP full mesh 💖 they can solve these corner cases by making sure that, within an AS, the advertisement of a new route is not translated into the withdraw of a former route. main character energy for stability 👑

and yes, advertising the best external route means an ASBR doesn't withdraw a previously advertised (EBGP) path when it receives an additional, preferred path over an IBGP session. and advertising the best intra-cluster route means an RR doesn't withdraw a previously advertised (IBGP) path to its non-clients (e.g., other RRs in a mesh of RR) when it receives a new, preferred path over an IBGP session. iconic 🦋

Acknowledgments 🫶

the authors wanted to thank Olivier Bonaventure, Pradosh Mohapatra, Job Snijders, John Heasley, and Christopher Morrow for their useful comments. huge thank you to them, they really helped 🌷

Authors' Addresses 💌

the people who wrote this, with their contact info:

Pierre Francois (editor)
Individual Contributor

Email: pfrpfr@gmail.com


Bruno Decraene (editor)
Orange

Email: bruno.decraene@orange.com


Cristel Pelsser
Strasbourg University

Email: pelsser@unistra.fr


Keyur Patel
Arrcus, Inc.

Email: keyur@arrcus.com


Clarence Filsfils
Cisco Systems

Email: cfilsfil@cisco.com
📦 where this page comes from