RFC 4724

Graceful Restart Mechanism for BGP

📅 published January 2007, from ietf

RFC 4724: Graceful Restart Mechanism for BGP 🦋

Okay bestie, here's the header tea ☕. This one came out of the Network Working Group in January 2007 as a Standards Track doc. S. Sangli and E. Chen wrote it from Cisco Systems, and R. Fernando, J. Scudder, and Y. Rekhter wrote it from Juniper Networks. It doesn't list anything it obsoletes or updates, so it's not replacing or patching any earlier RFC 🫶.

Status of This Memo ✨

This document specifies an Internet standards track protocol for the Internet community, and it asks for discussion and suggestions for improvements. For the standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited 🌸.

Copyright Notice 💜

Abstract 🧚

Okay so this doc is all about a mechanism for BGP that cuts down on the routing mess caused when BGP restarts 🫧. It introduces an End-of-RIB marker that can be used to signal routing convergence info. It also adds a brand new BGP capability called the "Graceful Restart Capability", which lets a BGP speaker say it's able to keep its forwarding state alive during a BGP restart 💅. Last but not least, it lays out procedures for temporarily holding onto routing information when a TCP session goes down and comes back up 🎀.

The mechanisms here apply to every router, whether it can preserve forwarding state during a BGP restart or can't. The ones that can't only need to implement a subset of what's described here 🌈.

Table of Contents 📋

1. Introduction ....................................................2
   1.1. Specification of Requirements ..............................2
2. Marker for End-of-RIB ...........................................3
3. Graceful Restart Capability .....................................3
4. Operation .......................................................6
   4.1. Procedures for the Restarting Speaker ......................6
   4.2. Procedures for the Receiving Speaker .......................7
5. Changes to BGP Finite State Machine .............................9
6. Deployment Considerations ......................................11
7. Security Considerations ........................................12
8. Acknowledgments ................................................13
9. IANA Considerations ............................................13
10. References ....................................................13
   10.1. Normative References .....................................13
   10.2. Informative References ...................................13

1. Introduction ✨

Okay, picture this 🙈. Usually when BGP on a router restarts, every one of its BGP peers notices the session went down and then came back up. That "down/up" wobble is a routing flap 😭, and it kicks off route re-computation, a fresh batch of BGP routing updates, and unnecessary churn in the forwarding tables. It can spread across multiple routing domains too. Flaps like this can cause temporary forwarding blackholes and/or temporary forwarding loops, they eat up control plane resources on the routers they hit, and overall they're just bad news for network performance 💀.

So this doc is the fix 🔥. It defines an End-of-RIB marker to convey routing convergence information. It creates the "Graceful Restart Capability" so a BGP speaker can announce it's able to preserve forwarding state during a restart. And it outlines how to temporarily keep routing information across a TCP session termination and re-establishment 🦋.

1.1. Specification of Requirements 📝

The capitalised key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are the spec-speak in this document, and they're meant to be read the way RFC 2119 [RFC2119] describes 🧐.

2. Marker for End-of-RIB 🌸

The End-of-RIB marker is a special UPDATE message with no reachable Network Layer Reachability Information (NLRI) and an empty withdrawn NLRI 🫧. A BGP speaker sends it to its peer to say "hey, I'm done with my initial routing update" once the session is established.

For the IPv4 unicast address family, the marker is an UPDATE message with the minimum length [BGP-4]. For any other address family, it's an UPDATE message that carries only the MP_UNREACH_NLRI attribute [BGP-MP] with no withdrawn routes for that <AFI, SAFI> 💅.

Even though this marker was made for BGP graceful restart, sending one when the initial update finishes is useful for routing convergence in general, so that practice is recommended 🎀.

Also, it's really helpful for convergence if a BGP speaker can tell its peer up front that it'll generate the End-of-RIB marker. That's true no matter whether the speaker can preserve its forwarding state during a restart 🥹. You can do this with the Graceful Restart Capability, which is coming up in the next section ✨.

3. Graceful Restart Capability 👑

The Graceful Restart Capability is a new BGP capability [BGP-CAP]. A BGP speaker uses it to say it can preserve its forwarding state during a BGP restart. It can also tell its peer it plans to generate the End-of-RIB marker once its initial routing updates are done 🧚.

Here's how the capability is laid out:

   Capability code: 64

   Capability length: variable

   Capability value: Consists of the "Restart Flags" field, "Restart
   Time" field, and 0 to 63 of the tuples <AFI, SAFI, Flags for
   address family> as follows:

      +--------------------------------------------------+
      | Restart Flags (4 bits)                           |
      +--------------------------------------------------+
      | Restart Time in seconds (12 bits)                |
      +--------------------------------------------------+
      | Address Family Identifier (16 bits)              |
      +--------------------------------------------------+
      | Subsequent Address Family Identifier (8 bits)    |
      +--------------------------------------------------+
      | Flags for Address Family (8 bits)                |
      +--------------------------------------------------+
      | ...                                              |
      +--------------------------------------------------+
      | Address Family Identifier (16 bits)              |
      +--------------------------------------------------+
      | Subsequent Address Family Identifier (8 bits)    |
      +--------------------------------------------------+
      | Flags for Address Family (8 bits)                |
      +--------------------------------------------------+

Now let's break down each field 🔍.

Restart Flags: This field holds bit flags related to restart 🚩.

    0 1 2 3
   +-+-+-+-+
   |R|Resv.|
   +-+-+-+-+

The most significant bit is the Restart State (R) bit. It can be used to avoid a possible deadlock caused by waiting for the End-of-RIB marker when multiple BGP speakers that peer with each other restart at the same time 🙃. When it's set (value 1), it means the BGP speaker has restarted, and its peer MUST NOT wait for the End-of-RIB marker from that speaker before advertising routing information to it.

The remaining bits are reserved, and they MUST be set to zero by the sender and ignored by the receiver 🤫.

Restart Time: This is the estimated time, in seconds, that the BGP session will take to be re-established after a restart ⏱️. A peer can use it to speed up routing convergence in case the BGP speaker never comes back after a restart.

Address Family Identifier (AFI), Subsequent Address Family Identifier (SAFI): The AFI and SAFI, taken together, show that Graceful Restart is supported for routes advertised with that same AFI and SAFI 🌈. Routes can be explicitly tied to a specific AFI and SAFI using the encoding of [BGP-MP], or they can be implicitly tied to <AFI=IPv4, SAFI=Unicast> if they use the encoding of [BGP-4].

Flags for Address Family: This field holds bit flags for routes that were advertised with the given AFI and SAFI 🍓.

    0 1 2 3 4 5 6 7
   +-+-+-+-+-+-+-+-+
   |F|   Reserved  |
   +-+-+-+-+-+-+-+-+

The most significant bit is the Forwarding State (F) bit. It can be used to show whether the forwarding state for routes advertised with the given AFI and SAFI was actually preserved during the previous BGP restart 🫶. When it's set (value 1), it means the forwarding state was preserved.

The remaining bits are reserved, and they MUST be set to zero by the sender and ignored by the receiver 🙈.

💅 Heads up: if a sender includes no <AFI, SAFI> at all in the capability, that means the sender can't preserve its forwarding state during a BGP restart. It does still support the procedures for the Receiving Speaker (as defined in Section 4.2 of this document). In that case, the "Restart Time" value the sender advertises doesn't matter.

A BGP speaker MUST NOT include more than one instance of the Graceful Restart Capability in the capability advertisement [BGP-CAP]. If more than one instance does show up anyway, the receiver of the advertisement MUST ignore all but the last instance 🥴.

Including <AFI=IPv4, SAFI=unicast> in the Graceful Restart Capability does not mean the IPv4 unicast routing information has to be carried using the BGP multiprotocol extensions [BGP-MP]. It could be carried in the NLRI field of the BGP UPDATE message instead 💖.

4. Operation ✨

okay so a BGP speaker MAY advertise the Graceful Restart Capability for an address family to its peer if it can hang onto its forwarding state for that family when BGP restarts 🔥 but here's the thing: even if a speaker can't preserve forwarding state for any address family during a restart, it's still recommended that it advertises the capability anyway 🫶 (remember, you do this by leaving out any <AFI, SAFI> in the advertised capability). there are two reasons for that. first, it's a heads up that the speaker will send an End-of-RIB marker once its initial routing updates are done, which helps convergence in general 💅 second, it tells a peer "hey, I'm down to support you if you want to do a graceful restart" 💖

The End-of-RIB marker MUST be sent by a BGP speaker to its peer once it completes the initial routing update (including the case when there is no update to send) for an address family after the BGP session is established. no skipping this one bestie 🙈

It is noted that the normal BGP procedures MUST be followed when the TCP session terminates due to the sending or receiving of a BGP NOTIFICATION message. so if a NOTIFICATION is involved, the regular BGP rules still apply, no graceful restart shortcuts 💀

A suggested default for the Restart Time is a value less than or equal to the HOLDTIME carried in the OPEN. 🌸

In the following sections, "Restarting Speaker" refers to a router whose BGP has restarted, and "Receiving Speaker" refers to a router that peers with the restarting speaker. so the Restarting Speaker is the one that went down and came back, and the Receiving Speaker is its buddy on the other end 🦋

Consider that the Graceful Restart Capability for an address family is advertised by the Restarting Speaker, and is understood by the Receiving Speaker, and a BGP session between them is established. the next sections spell out the procedures that MUST be followed by the Restarting Speaker and the Receiving Speaker once the Restarting Speaker restarts. it's giving choreography 🧚

4.1. Procedures for the Restarting Speaker

When the Restarting Speaker restarts, it MUST retain, if possible, the forwarding state for the BGP routes in the Loc-RIB and MUST mark them as stale. it MUST NOT differentiate between stale and other information during forwarding. so stale routes still get forwarded, no judgement 🫠

To re-establish the session with its peer, the Restarting Speaker MUST set the "Restart State" bit in the Graceful Restart Capability of the OPEN message. unless configuration says otherwise, the "Forwarding State" bit for an address family in the capability can be set only if the forwarding state has indeed been preserved for that address family during the restart. so no lying about having your forwarding state 🙅‍♀️✨

Once the session between the Restarting Speaker and the Receiving Speaker is re-established, the Restarting Speaker will receive and process BGP messages from its peers. however, it MUST defer route selection for an address family until it either (a) receives the End-of-RIB marker from all its peers (excluding the ones with the "Restart State" bit set in the received capability and excluding the ones that do not advertise the graceful restart capability) or (b) the Selection_Deferral_Timer referred to below has expired. it's noted that prior to route selection, the speaker has no routes to advertise to its peers and no routes to update the forwarding state. so it's literally waiting around with nothing to say yet 🧸

In situations where both Interior Gateway Protocol (IGP) and BGP have restarted, it might be advantageous to wait for IGP to converge before the BGP speaker performs route selection. 🌈

After the BGP speaker performs route selection, the forwarding state of the speaker MUST be updated and any previously marked stale information MUST be removed. the Adj-RIB-Out can then be advertised to its peers. once the initial update is complete for an address family (including the case that there is no routing update to send), the End-of-RIB marker MUST be sent. it's the glow-up finale 👑

To put an upper bound on the amount of time a router defers its route selection, an implementation MUST support a (configurable) timer that imposes this upper bound. this timer is called the "Selection_Deferral_Timer". the value of this timer should be large enough to give all the peers of the Restarting Speaker enough time to send all the routes to the Restarting Speaker. ⏰

If one wants to apply graceful restart only when the restart is planned (as opposed to both planned and unplanned restart), then one way to accomplish this would be to set the Forwarding State bit to 1 after a planned restart, and to 0 in all other cases. other approaches to accomplish this are outside the scope of this document. 🎀

4.2. Procedures for the Receiving Speaker

When the Restarting Speaker restarts, the Receiving Speaker may or may not detect the termination of the TCP session with the Restarting Speaker. that depends on the underlying TCP implementation, whether or not [BGP-AUTH] is in use, and the specific circumstances of the restart. if it does not detect the termination of the old TCP session and still considers the BGP session as being established, it MUST treat the subsequent open connection from the peer as an indication of the termination of the old TCP session and act accordingly (when the Graceful Restart Capability has been received from the peer). See Section 8 for a description of this behavior in terms of the BGP finite state machine. 🦋

"Acting accordingly" in this context means that the previous TCP session MUST be closed, and the new one retained. note that this behavior differs from the default behavior, as specified in [BGP-4], Section 6.8. since the previous connection is considered to be terminated, no NOTIFICATION message should be sent -- the previous TCP session is simply closed. 🫶

When the Receiving Speaker detects termination of the TCP session for a BGP session with a peer that has advertised the Graceful Restart Capability, it MUST retain the routes received from the peer for all the address families that were previously received in the Graceful Restart Capability and MUST mark them as stale routing information. to deal with possible consecutive restarts, a route (from the peer) previously marked as stale MUST be deleted. the router MUST NOT differentiate between stale and other routing information during forwarding. 🌸

In re-establishing the session, the "Restart State" bit in the Graceful Restart Capability of the OPEN message sent by the Receiving Speaker MUST NOT be set unless the Receiving Speaker has restarted. the presence and the setting of the "Forwarding State" bit for an address family depend upon the actual forwarding state and configuration. 💜

If the session does not get re-established within the "Restart Time" that the peer advertised previously, the Receiving Speaker MUST delete all the stale routes from the peer that it is retaining. ⏳

A BGP speaker could have some way of determining whether its peer's forwarding state is still viable, for example through Bidirectional Forwarding Detection [BFD] or through monitoring layer two information. specifics of such mechanisms are beyond the scope of this document. if it determines that its peer's forwarding state is not viable prior to the re-establishment of the session, the speaker MAY delete all the stale routes from the peer that it is retaining. 🔍

Once the session is re-established, if the "Forwarding State" bit for a specific address family is not set in the newly received Graceful Restart Capability, or if a specific address family is not included in the newly received Graceful Restart Capability, or if the Graceful Restart Capability is not received in the re-established session at all, then the Receiving Speaker MUST immediately remove all the stale routes from the peer that it is retaining for that address family. no vibes-based waiting here 💀

The Receiving Speaker MUST send the End-of-RIB marker once it completes the initial update for an address family (including the case that it has no routes to send) to the peer. 🏁

The Receiving Speaker MUST replace the stale routes by the routing updates received from the peer. once the End-of-RIB marker for an address family is received from the peer, it MUST immediately remove any routes from the peer that are still marked as stale for that address family. ✨

To put an upper bound on the amount of time a router retains the stale routes, an implementation MAY support a (configurable) timer that imposes this upper bound. 🎀

5. Changes to BGP Finite State Machine ✨

okay so, like we said back in "Procedures for the Receiving Speaker" 💅, this spec actually tweaks the BGP finite state machine 🦋. Here are the exact changes to [BGP-4], Section 8.2.2 🌸

In the Idle state, make these changes:

Replace this text:

   - initializes all BGP resources for the peer connection,

with

   - initializes all BGP resources for the peer connection, other
     than those resources required in order to retain routes
     according to section "Procedures for the Receiving Speaker" of
     this (Graceful Restart) specification,

In the Established state, make these changes too 🫶

Replace this text:

     In response to an indication that the TCP connection is
     successfully established (Event 16 or Event 17), the second
     connection SHALL be tracked until it sends an OPEN message.

with

     If the Graceful Restart Capability with one or more AFIs/SAFIs
     has not been received for the session, then in response to an
     indication that a TCP connection is successfully established
     (Event 16 or Event 17), the second connection SHALL be tracked
     until it sends an OPEN message.

     However, if the Graceful Restart Capability with one or more
     AFIs/SAFIs has been received for the session, then in response
     to Event 16 or Event 17 the local system:

      - retains all routes associated with this connection according
        to section "Procedures for the Receiving Speaker" of this
        (Graceful Restart) specification,

      - releases all other BGP resources,

      - drops the TCP connection associated with the ESTABLISHED
        session,

      - initializes all BGP resources for the peer connection, other
        than those required in order to retain routes according to
        section "Procedures for the Receiving Speaker" of this
        specification,

      - sets ConnectRetryCounter to zero,

      - starts the ConnectRetryTimer with the initial value, and

      - changes its state to Connect.

Replace this text:

   If the local system receives a NOTIFICATION message (Event 24 or
   Event 25), or a TcpConnectionFails (Event 18) from the underlying
   TCP, the local system:

      - sets the ConnectRetryTimer to zero,

      - deletes all routes associated with this connection,

      - releases all the BGP resources,

      - drops the TCP connection,

      - increments the ConnectRetryCounter by 1,

      - changes its state to Idle.

with

   If the local system receives a NOTIFICATION message (Event 24 or
   Event 25), or if the local system receives a TcpConnectionFails
   (Event 18) from the underlying TCP and the Graceful Restart
   capability with one or more AFIs/SAFIs has not been received for
   the session, the local system:

      - sets the ConnectRetryTimer to zero,

      - deletes all routes associated with this connection,

      - releases all the BGP resources,

      - drops the TCP connection,

      - increments the ConnectRetryCounter by 1, and

      - changes its state to Idle.

   However, if the local system receives a TcpConnectionFails (Event
   18) from the underlying TCP, and the Graceful Restart Capability
   with one or more AFIs/SAFIs has been received for the session, the
   local system:

      - sets the ConnectRetryTimer to zero,

      - retains all routes associated with this connection according
        to section "Procedures for the Receiving Speaker" of this
        (Graceful Restart) specification,

      - releases all other BGP resources,

      - drops the TCP connection,

      - increments the ConnectRetryCounter by 1, and

      - changes its state to Idle.

6. Deployment Considerations 🎀

okay real talk bestie, these procedures help cut down on the effect of routing flaps 🦋 but there are still some spicy edge cases 🙈. When a BGP Graceful Restart-capable router restarts, or when it restarts without preserving its forwarding state (for example, after a power failure), there's a potential for transient routing loops or blackholes in the network if routing information changes before the routers involved finish their routing updates and convergence. 💀

Also, depending on the network topology, if not all IBGP speakers are Graceful Restart capable, there could be more exposure to transient routing loops or blackholes whenever the Graceful Restart procedures get exercised 😭

The Restart Time, the upper bound for retaining routes, and the upper bound for deferring route selection might need tuning as more deployment experience comes in 🌈

Finally, the benefits of deploying BGP Graceful Restart in an Autonomous System (AS) are reduced if its IGPs and BGP are tightly coupled (meaning BGP and the IGPs would both restart) and the IGPs have no similar Graceful Restart Capability, compared to a scenario where the IGPs do have a similar Graceful Restart Capability ✨

7. Security Considerations 🔥

okay this part is serious, so pay attention 👑

Since this proposal lets a new connection cause an old one to be terminated, it might seem like it opens the door to denial of service attacks. However, unauthenticated BGP is already known to be vulnerable to denial of service through attacks on the TCP transport 😬 The TCP transport is commonly protected through the use of [BGP-AUTH]. That authentication equally protects against denial of service through spurious new connections 🛡️

If an attacker can successfully open a TCP connection while impersonating a legitimate peer, the attacker's connection replaces the legitimate one, which could let the attacker advertise bogus routes. We note, however, that the window for such a route insertion attack is small. Through normal operation of the protocol, the legitimate peer would open a new connection, which in turn causes the attacker's connection to be terminated. So this attack devolves into a form of denial of service 💀

It is therefore concluded that this proposal does not change the underlying security model (and issues) of BGP-4 🌸

We also note that implementations may allow use of graceful restart to be controlled by configuration. If graceful restart is not enabled, the underlying security model of BGP-4 is naturally unchanged 🧚

8. Acknowledgments 🫶

The authors would like to thank Bruce Cole, Lars Eggert, Bill Fenner, Eric Gray, Jeffrey Haas, Sam Hartman, Alvaro Retana, Pekka Savola Naiming Shen, Satinder Singh, Mark Townsley, David Ward, Shane Wright, and Alex Zinin for their review and comments 💖

9. IANA Considerations 🍓

This document defines a new BGP capability, the Graceful Restart Capability. The Capability Code for Graceful Restart Capability is 64 🌈

10. References 📚

10.1. Normative References

[BGP-4]     Rekhter, Y., Li, T., and S. Hares, "A Border Gateway
            Protocol 4 (BGP-4)", RFC 4271, January 2006.

[BGP-MP]    Bates, T., Rekhter, Y., Chandra, R., and D. Katz,
            "Multiprotocol Extensions for BGP-4", RFC 2858, June
            2000.

[BGP-CAP]   Chandra, R. and J. Scudder, "Capabilities Advertisement
            with BGP-4", RFC 3392, November 2002.

[BGP-AUTH]  Heffernan, A., "Protection of BGP Sessions via the TCP
            MD5 Signature Option", RFC 2385, August 1998.

[RFC2119]   Bradner, S., "Key words for use in RFCs to Indicate
            Requirement Levels", BCP 14, RFC 2119, March 1997.

[IANA-AFI]  http://www.iana.org/assignments/address-family-numbers

[IANA-SAFI] http://www.iana.org/assignments/safi-namespace

10.2. Informative References

[BFD]       Katz, D. and D. Ward, "Bidirectional Forwarding
            Detection", Work in Progress.

Authors' Addresses 🧑‍💻

Srihari R.  Sangli
Cisco Systems, Inc.

EMail: rsrihari@cisco.com


Yakov Rekhter
Juniper Networks, Inc.

EMail: yakov@juniper.net


Rex Fernando
Juniper Networks, Inc.

EMail: rex@juniper.net


John G.  Scudder
Juniper Networks, Inc.

EMail: jgs@juniper.net


Enke Chen
Cisco Systems, Inc.

EMail:  enkechen@cisco.com

Full Copyright Statement 📜

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🔐

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document, or the extent to which any license under such rights might or might not be available. It also doesn't represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.

Acknowledgement 💖

Funding for the RFC Editor function is currently provided by the Internet Society 🌸

📦 where this page comes from

ietf: rfc4724 2007-01
💖 girlified by claude-haiku-5-5 on 2026-10-16: a language model's rewrite of the original, under its terms