RFC 8538

Notification Message Support for BGP Graceful Restart

📅 published March 2019, from ietf

Notification Message Support for BGP Graceful Restart ✨

Okay bestie, let's get into RFC 8538 🦋 It's from the IETF (Internet Engineering Task Force), it's a Standards Track doc dated March 2019, and it was written by K. Patel (Arrcus), R. Fernando (Cisco Systems), J. Scudder, and J. Haas (Juniper Networks) 💅 It updates RFC 4724 (that's the Graceful Restart one), and there's no Obsoletes line, so it doesn't replace anything. It's giving official IETF energy 👑

Abstract ✨

So the BGP Graceful Restart mechanism from RFC 4724 only works for BGP messages that aren't NOTIFICATION messages 🙈 This doc updates RFC 4724 by adding an extension that lets Graceful Restart procedures run when a BGP speaker gets a NOTIFICATION message or its Hold Time expires 🔥 It also creates a new subcode for BGP Cease NOTIFICATION messages. That subcode asks for a full session restart instead of a Graceful Restart, which is so iconic 💜

Status of This Memo 📜

This is an Internet Standards Track document.

It's a product of the Internet Engineering Task Force (IETF), so it represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). For more on Internet Standards, check Section 2 of RFC 7841 🌸

For the current status of this document, any errata, and how to send feedback, go to https://www.rfc-editor.org/info/rfc8538 🎀

Copyright Notice 🌈

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, since they explain your rights and restrictions for this document. Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 🫶

Table of Contents 🧚

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
  1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
2.  Modifications to BGP Graceful Restart Capability  . . . . . .   3
3.  BGP Hard Reset Subcode  . . . . . . . . . . . . . . . . . . .   4
  3.1.  Sending a Hard Reset  . . . . . . . . . . . . . . . . . .   4
  3.2.  Receiving a Hard Reset  . . . . . . . . . . . . . . . . .   4
4.  Operation . . . . . . . . . . . . . . . . . . . . . . . . . .   5
  4.1.  Rules for the Receiving Speaker . . . . . . . . . . . . .   6
5.  Use of Hard Reset . . . . . . . . . . . . . . . . . . . . . .   7
  5.1.  When to Send a Hard Reset . . . . . . . . . . . . . . . .   7
  5.2.  Interaction with Other Specifications . . . . . . . . . .   7
6.  Management Considerations . . . . . . . . . . . . . . . . . .   8
7.  Operational Considerations  . . . . . . . . . . . . . . . . .   8
8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
9.  Security Considerations . . . . . . . . . . . . . . . . . . .   9
10. References  . . . . . . . . . . . . . . . . . . . . . . . . .   9
  10.1.  Normative References . . . . . . . . . . . . . . . . . .   9
  10.2.  Informative References . . . . . . . . . . . . . . . . .   9
Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  10
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  10

1. Introduction ✨

Okay so, for lots of error types, BGP has to send a NOTIFICATION message and reset the peering session to deal with the problem 😭 The Graceful Restart mechanism in [RFC4724] says normal BGP procedures from [RFC4271] have to be followed whenever a NOTIFICATION message is sent or received. This doc adds an extension to BGP Graceful Restart so that Graceful Restart procedures can run when the BGP speaker gets a NOTIFICATION message or its Hold Time expires. That way the speaker can avoid flapping reachability and keep forwarding while it restarts the session to handle errors found in BGP 🦋

Here's the big picture 🔥 When a BGP session resets, both speakers act as "Receiving Speakers" under [RFC4724], which means they keep each other's routes. The same goes for HOLDTIME expiration. You can defeat all of this by sending a BGP Cease NOTIFICATION message with the Hard Reset subcode, and if a Hard Reset is used, a full session reset happens instead 💀

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 🎀

2. Modifications to BGP Graceful Restart Capability 💜

The BGP Graceful Restart Capability gets augmented so it can signal Graceful Restart support for BGP NOTIFICATION messages. The Restart Flags field is augmented like this (it follows the diagram in Section 3 of [RFC4724]) 🧚

   Restart Flags:

         This field contains bit flags relating to restart.

                0 1 2 3
               +-+-+-+-+
               |R|N|   |
               +-+-+-+-+

The most significant bit is the Restart State ("R") bit, defined in [RFC4724] ✨

The second most significant bit is the Graceful Notification ("N") bit, and this doc defines it. It shows Graceful Restart support for BGP NOTIFICATION messages. A BGP speaker says it supports the procedures in this document by advertising a Graceful Restart Capability with its "N" bit set (value 1) 🌸

If a BGP speaker that already advertised a set of Graceful Restart parameters opens a new session with different parameters, the new parameters apply once the session has moved into ESTABLISHED state 🍓

3. BGP Hard Reset Subcode 🔥

This doc defines a new subcode for BGP Cease NOTIFICATION messages, called the Hard Reset subcode. Its value is discussed in Section 8. In this document, a BGP Cease NOTIFICATION message with the Hard Reset subcode is called a "Hard Reset message", or just a "Hard Reset" 👑

When two peers have exchanged the "N" bit, the NOTIFICATION messages other than Hard Reset messages are called "Graceful", since they invoke Graceful Restart semantics 🫶

3.1. Sending a Hard Reset 💅

When the "N" bit has been exchanged, a Hard Reset message tells the peer that the session is to be fully terminated 💖

When you send a Hard Reset, the data portion of the NOTIFICATION message is encoded like this:

       +--------+--------+--------
       | ErrCode| Subcode| Data
       +--------+--------+--------

ErrCode is a BGP Error Code (as documented in the IANA "BGP Error (Notification) Codes" registry) that shows the reason for the Hard Reset. Subcode is a BGP Error Subcode (as documented in the IANA "BGP Error Subcodes" registry) that fits the ErrCode. Data is also whatever fits the ErrCode and Subcode. Basically, the Hard Reset wraps another NOTIFICATION message inside its data portion 🌈

3.2. Receiving a Hard Reset ⭐

Whenever a BGP speaker receives a Hard Reset, the speaker MUST terminate the BGP session following the standard procedures in [RFC4271] 😭

4. Operation 🎀

A BGP speaker that's willing to receive and send BGP NOTIFICATION messages under the procedures of this document MUST advertise the "N" bit using the Graceful Restart Capability defined in [RFC4724] 💖

When that kind of BGP speaker has received the "N" bit from its peer, and it gets a BGP NOTIFICATION message from that peer that isn't a Hard Reset, it MUST follow the rules for the Receiving Speaker in Section 4.1. The BGP speaker that generates the BGP NOTIFICATION message MUST also follow the rules for the Receiving Speaker 🔥

When a BGP speaker resets its session because of a HOLDTIME expiry, it should generate the relevant BGP NOTIFICATION message as mentioned in [RFC4271], but then it MUST follow the rules for the Receiving Speaker in Section 4.1 🧚

A BGP speaker SHOULD NOT send a Hard Reset to a peer it hasn't received the "N" bit from. We do note that if it did, the effect would still be what we want, because according to [RFC4271] and [RFC4724], any NOTIFICATION message, whether recognized or not, results in a session reset. So the only downside of sending a Hard Reset to a peer that hasn't said it complies with this spec is that the peer wouldn't be able to properly log the related info 🙈

Once the session is re-established, both BGP speakers SHOULD set their Forwarding State bit to 1. If the Forwarding State bit isn't set, then under the procedures in Section 4.2 of [RFC4724] the relevant routes will be flushed, and that defeats the whole point of this spec 😭🫶

4.1. Rules for the Receiving Speaker 🎀

Okay so, Section 4.2 of [RFC4724] already has the rules for the Receiving Speaker, and this doc is giving them a glow-up 💅. Here's what changes.

First up, the sentence "To deal with possible consecutive restarts, a route (from the peer) previously marked as stale MUST be deleted" only applies when the "N" bit has not been exchanged with the peer 🙈. Here's the before and after:

OLD:

When the Receiving Speaker detects termination of the TCP
session for a BGP session with a peer that has advertised the
Graceful Restart Capability, it MUST retain the routes received
from the peer for all the address families that were previously
received in the Graceful Restart Capability and MUST mark them
as stale routing information.  To deal with possible consecutive
restarts, a route (from the peer) previously marked as stale
MUST be deleted.  The router MUST NOT differentiate between
stale and other routing information during forwarding.

NEW:

When the Receiving Speaker detects termination of the TCP
session for a BGP session with a peer that has advertised the
Graceful Restart Capability, it MUST retain the routes received
from the peer for all the address families that were previously
received in the Graceful Restart Capability and MUST mark them
as stale routing information.  The router MUST NOT differentiate
between stale and other routing information during forwarding.
If the "N" bit has not been exchanged with the peer, then to
deal with possible consecutive restarts, a route (from the peer)
previously marked as stale MUST be deleted.

The stale timer also gets a formal name and becomes mandatory ✨ Look at this glow-up:

OLD:

To put an upper bound on the amount of time a router retains the
stale routes, an implementation MAY support a (configurable)
timer that imposes this upper bound.

NEW:

To put an upper bound on the amount of time a router retains the
stale routes, an implementation MUST support a (configurable)
timer, called the "stale timer", that imposes this upper bound.
A suggested default value for the stale timer is 180 seconds.
An implementation MAY provide the option to disable the timer
(i.e., to provide an infinite retention time) but MUST NOT do so
by default.

5. Use of Hard Reset ✨

5.1. When to Send a Hard Reset 🧊

Sending a Hard Reset is an implementation-specific decision, but we've got some advice to share 💖. Lots of Cease NOTIFICATION subcodes mean the session is ending for good or for a long time, not just a blip. So using Hard Reset with them makes sense 🙌. As of publication of this document, subcodes 1-9 have been defined for Cease. The table below lists each one with its suggested behavior.

+-------+------------------------------------+----------------------+
| Value |                Name                |  Suggested Behavior  |
+-------+------------------------------------+----------------------+
|   1   | Maximum Number of Prefixes Reached |      Hard Reset      |
|   2   |      Administrative Shutdown       |      Hard Reset      |
|   3   |         Peer De-configured         |      Hard Reset      |
|   4   |        Administrative Reset        | Provide user control |
|   5   |        Connection Rejected         |    Graceful Cease    |
|   6   |     Other Configuration Change     |    Graceful Cease    |
|   7   |  Connection Collision Resolution   |    Graceful Cease    |
|   8   |          Out of Resources          |    Graceful Cease    |
|   9   |             Hard Reset             |      Hard Reset      |
+-------+------------------------------------+----------------------+

These are just suggestions, not requirements 🫶. BGP implementations don't always map internal states to NOTIFICATION codes and subcodes perfectly, and that's just how it is 🤷‍♀️. The big guiding principle for implementors: if there's no realistic hope that forwarding can continue or the session will come back within the deadline, Hard Reset should be used.

For all NOTIFICATION codes other than Cease, using Hard Reset does not appear to be indicated 🙅‍♀️.

5.2. Interaction with Other Specifications 🔗

"BGP Administrative Shutdown Communication" [RFC8203] lets you use the data portion of the Administrative Shutdown or Administrative Reset subcodes to send a short message 💌. When [RFC8203] is used together with Hard Reset, the subcode of the outermost Cease MUST be Hard Reset, with the Administrative Shutdown or Administrative Reset subcodes encapsulated within 📦. The encapsulated message MUST subsequently be processed according to [RFC8203].

6. Management Considerations 📊

When you report a Hard Reset to network management, the error code and subcode reported MUST be Cease and Hard Reset, respectively. If the network management layer you're using permits it, the information carried in the Data portion SHOULD be reported as well 📝.

7. Operational Considerations ⚠️

Note that long (or infinite) retention time may cause operational issues and should be enabled with care. Be careful with this one, bestie 😭.

8. IANA Considerations 🧚

IANA has assigned subcode 9 ("Hard Reset") in the "BGP Cease NOTIFICATION message subcodes" registry ✨.

IANA has also created a sub-registry called "BGP Graceful Restart Flags" under the "Border Gateway Protocol (BGP) Parameters" registry 🗂️. The registration procedure is Standards Action [RFC8126]. This document and [RFC4724] are listed as references. The initial values look like this:

      +--------------+---------------+------------+-----------+
      | Bit Position |      Name     | Short Name | Reference |
      +--------------+---------------+------------+-----------+
      |      0       | Restart State |     R      |  RFC 4724 |
      |      1       |  Notification |     N      |  RFC 8538 |
      |     2-3      |   Unassigned  |            |           |
      +--------------+---------------+------------+-----------+

IANA has created another sub-registry, called "BGP Graceful Restart Flags for Address Family", under the same "Border Gateway Protocol (BGP) Parameters" registry 🌈. The registration procedure is Standards Action. This document and [RFC4724] are listed as references. The initial values are:

    +--------------+------------------+------------+-----------+
    | Bit Position |       Name       | Short Name | Reference |
    +--------------+------------------+------------+-----------+
    |      0       | Forwarding State |     F      |  RFC 4724 |
    |     1-7      |    Unassigned    |            |           |
    +--------------+------------------+------------+-----------+

9. Security Considerations 🔒

Okay, this section is serious business, so listen up 🚨. This specification doesn't change the basic security model that's built into [RFC4724], with one exception: the protection against repeated resets is relaxed 😬. That opens the door to an attacker who could use repeated session resets to keep stale routes from ever being deleted. To shut that down, we make the stale timer mandatory. Honestly, it's already everywhere in practice anyway 💅. As for denial-of-service defense, [RFC4724] might be said to help by making the control plane more resilient, and this extension may modestly increase that resilience 🛡️. But there are enough confounding and deployment-specific factors that no general claims can be made, so don't let anyone tell you otherwise 🙅‍♀️.

10. References 📚

10.1. Normative References

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
           Border Gateway Protocol 4 (BGP-4)", RFC 4271,
           DOI 10.17487/RFC4271, January 2006,
           <https://www.rfc-editor.org/info/rfc4271>.

[RFC4724]  Sangli, S., Chen, E., Fernando, R., Scudder, J., and Y.
           Rekhter, "Graceful Restart Mechanism for BGP", RFC 4724,
           DOI 10.17487/RFC4724, January 2007,
           <https://www.rfc-editor.org/info/rfc4724>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC8203]  Snijders, J., Heitz, J., and J. Scudder, "BGP
           Administrative Shutdown Communication", RFC 8203,
           DOI 10.17487/RFC8203, July 2017,
           <https://www.rfc-editor.org/info/rfc8203>.

10.2. Informative References

[RFC8126]  Cotton, M., Leiba, B., and T. Narten, "Guidelines for
           Writing an IANA Considerations Section in RFCs", BCP 26,
           RFC 8126, DOI 10.17487/RFC8126, June 2017,
           <https://www.rfc-editor.org/info/rfc8126>.

Acknowledgements 🫶

The authors want to thank Jim Uttaro for the suggestion 💜. They also want to thank Emmanuel Baccelli, Bruno Decraene, Chris Hall, Warren Kumari, Paul Mattes, Robert Raszuk, and Alvaro Retana for their reviews and comments 🌟.

Authors' Addresses 🌸

Keyur Patel
Arrcus

Email: keyur@arrcus.com


Rex Fernando
Cisco Systems
170 W. Tasman Drive
San Jose, CA  95134
United States of America

Email: rex@cisco.com


John Scudder
Juniper Networks
1194 N. Mathilda Ave
Sunnyvale, CA  94089
United States of America

Email: jgs@juniper.net


Jeff Haas
Juniper Networks
1194 N. Mathilda Ave
Sunnyvale, CA  94089
United States of America

Email: jhaas@juniper.net

📦 where this page comes from

ietf: rfc8538 2019-03
💖 girlified by claude-haiku-5-5 on 2026-10-18: a language model's rewrite of the original, under its terms