RFC 3478

Graceful Restart Mechanism for Label Distribution Protocol 🎀

okay bestie, this is RFC 3478 and it's giving main character energy ✨ It's a Standards Track doc from February 2003, written by M. Leelanivas and Y. Rekhter (Juniper Networks) and R. Aggarwal (Redback Networks). The header doesn't have an Obsoletes or Updates line, so it isn't replacing or amending anything listed there 💅

Status of this Memo 📜

So this doc is an Internet standards track protocol for the Internet community, and it's asking for discussion and suggestions for improvements 💖 For the actual standardization state, go check the current edition of "Internet Official Protocol Standards" (STD 1). And yes, distribution is unlimited, so share away 🦋

Copyright Notice ©️

The Internet Society holds the copyright (2003), and all rights are reserved 🔒

Abstract 🌸

Okay so the big idea is this: when a Label Switching Router (LSR) restarts its control plane, the MPLS traffic going through it can get messed up 😭 This mechanism helps stop that from happening, specifically when the thing restarting is its Label Distribution Protocol (LDP) component. It only works on LSRs that can keep their MPLS forwarding component alive during the restart ✨

Here's the thing though, it applies to all LSRs. Ones that can preserve forwarding state during an LDP restart, and ones that can't. The ones that can't only need to implement a subset of it 🫶 If those LSRs still do the subset, it doesn't fix the damage to their own MPLS traffic, but it cuts down the impact when their neighbors can keep forwarding state through the restart and run the full mechanism. Neighbor energy, honestly 🧚

The mechanism is also super minimal about what needs saving. It only assumes the actual MPLS forwarding state has to be preserved. None of the LDP-related state has to survive the restart 🍓

And the procedures here are for downstream unsolicited label distribution. Downstream on demand is for further study, so that's a "not yet" 🙈

Specification of Requirements 🔥

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be read as described in BCP 14, which is RFC 2119 [RFC2119] 👑

1. Motivation 💜

Quick shorthand alert 🚨 For brevity in this doc, "the control plane" just means "the LDP component of the control plane" ✨

Also, "MPLS forwarding state" means either a <incoming label -> (outgoing label, next hop)> mapping (the non-ingress case) or a <FEC->(outgoing label, next hop)> mapping (the ingress case) 🎀

So when an LSR could keep its MPLS forwarding state through a restart of its LDP component [LDP], we really want to avoid disturbing the LSPs running through it (the ones LDP set up). This doc calls the way to do that "LDP Graceful Restart" 🦋

Same deal as before, the mechanism applies to all LSRs, with or without the ability to keep forwarding state. The ones without only need a subset 💅 Doing that subset doesn't reduce the MPLS damage from their own restart, but it helps if their neighbors can keep forwarding state and implement the full mechanism 🫶

The minimalism is the vibe here too. Only the actual MPLS forwarding state gets preserved, and that's honestly the least amount of state you could save without disturbing LSPs crossing a restarting LSR. No LDP-related state has to survive ✨

Now the tricky scenario. Sometimes an LSR creates or keeps label bindings using more than just LDP. Think BGP or RSVP-TE 👀 If the LSR can restart those components on their own (say LDP restarts but BGP doesn't), then it has to keep track, across the restart, of which protocol assigned which labels 🔐

And yep, the downstream unsolicited thing shows up again. Those procedures are the only ones covered here, and downstream on demand is for further study 🙈

2. LDP Extension 🧚

An LSR says it can do LDP Graceful Restart (as defined here) by putting the Fault Tolerant (FT) Session TLV in the LDP Initialization message as an Optional Parameter. The format of that TLV lives in [FT-LDP] 🎀 The L (Learn from Network) flag MUST be set to 1, which tells everyone the procedures in this doc are in use. The other FT flags are set to 0 by the sender and ignored when received 💖

The value field of the FT Session TLV has two parts that this mechanism uses: the FT Reconnect Timeout and the Recovery Time ✨

The FT Reconnect Timeout is how long, in milliseconds, the sender would like the receiver to wait after the receiver notices LDP communication with the sender has failed. While it's waiting, the receiver SHOULD keep the MPLS forwarding state for the (already established) LSPs running over the link between the two. The FT Reconnect Timeout should be long enough for the sender's control plane to restart, specifically its LDP component, and get to the point where it can swap LDP messages with its neighbors 🌸

Setting the FT Reconnect Timeout to 0 means the sender won't keep its forwarding state across a restart. BUT the sender still supports the procedures in "Section 3.3, "Restart of LDP communication with a neighbor LSR" of this document," so it can still benefit if its neighbor keeps forwarding state through the restart 🫶

For a restarting LSR, the Recovery Time is how long, in milliseconds, it's willing to keep the MPLS forwarding state it saved across the restart. The clock starts from when the LSR sends the Initialization message carrying the FT Session TLV after restarting. Setting this to 0 means the MPLS forwarding state wasn't preserved across the restart, or if it was, it's no longer around 😭

The Recovery Time SHOULD be long enough for the neighboring LSR's to re-sync all the LSP's in a graceful way, without creating congestion in the LDP control plane 🦋

3. Operations ⚡

An LSR that supports what's described here tells its LDP neighbors about it by carrying the FT Session TLV in the LDP Initialization message. Simple, iconic 💅

This doc assumes that in certain situations, described in section 3.1.2, "Egress LSR", an LSR can preserve its IP forwarding state across a restart too, on top of its MPLS forwarding state. The procedures for keeping IP forwarding state across a restart are defined in [OSPF-RESTART], [ISIS-RESTART], and [BGP-RESTART] 🔥

3.1. Procedures for the restarting LSR 🔄

After an LSR restarts its control plane, it MUST check whether it managed to preserve its MPLS forwarding state from before the restart. If it didn't, the LSR sets the Recovery Time to 0 in the FT Session TLV it sends to its neighbors 💀

If the forwarding state was preserved, the LSR starts its internal timer, called the MPLS Forwarding State Holding timer (that timer's value SHOULD be configurable), and marks every MPLS forwarding state entry as "stale" 🌈 When the timer expires, any entries still marked stale SHOULD be deleted. The Recovery Time advertised in the FT Session TLV is set to the timer's current value at the moment the Initialization message carrying the FT Session TLV gets sent ✨

We say an LSR is in the process of restarting while the MPLS Forwarding State Holding timer has not expired. Once that timer expires, the LSR has completed its restart 🎉

The following procedures apply when an LSR is in the process of restarting.

3.1.1. Non-egress LSR 🦋

okay so this one is for LSRs that are NOT the egress 💅 first case: if the label in the newly received Mapping message is not an Implicit NULL, the LSR digs through its MPLS forwarding state looking for an entry whose outgoing label matches the label in the message AND whose next hop is one of the addresses (next hops) from the peer's Address message. found one? then it's no longer marked as stale ✨ and if that entry is <incoming label, (outgoing label, next hop)> (not the <FEC, (outgoing label, next hop)> kind), the LSR associates the incoming label from that entry with the FEC from the Label Mapping message and advertises <incoming label, FEC> to its neighbors via LDP 🧚 no incoming label on the entry, or no entry at all? it just follows the normal LDP procedures.

(quick note: this paragraph is the scenario where the restarting LSR is neither the egress nor a penultimate hop using penultimate hop popping for that LSP. it also covers the case where the restarting LSR is the ingress 🫶)

okay now the Implicit NULL case 🌸 if the label in the Mapping message is an Implicit NULL label, the LSR searches for an entry that indicates Label pop (means no outgoing label) and whose next hop is one of the addresses (next hops) from the peer's Address message. if it finds that, it stops marking the entry as stale, associates the incoming label from that entry with the FEC from the Label Mapping message from the neighbor, and advertises <incoming label, FEC> to its neighbors. no incoming label or no entry? normal LDP procedures, bestie.

(this paragraph describes the scenario where the restarting LSR is a penultimate hop for a particular LSP, and that LSP uses penultimate hop popping 💜)

this next bit is where it gets spicy 🌶️ the paragraph above assumes the restarting LSR generates the same label for all the LSPs that terminate on the same LSR (different from the restarting LSR) and for which it is a penultimate hop. if that's NOT the case, and it generates a unique label per each of those LSPs, then the LSR needs to preserve across the restart not just the <incoming label, (outgoing label, next hop)> mapping, but also the FEC associated with that mapping 😭 so it searches for an entry that (a) indicates Label pop (means no outgoing label), (b) has a next hop equal to one of the addresses (next hops) from the peer's Address message, and (c) has the same FEC as the one in the Label Mapping message. found it? same deal: not stale anymore, associate the incoming label with the FEC, advertise <incoming label, FEC> to neighbors. no incoming label or no entry, normal LDP procedures. periodt.

3.1.2. Egress LSR 👑

if an LSR figures out it's the egress for a particular FEC, and it's configured to generate a non-NULL label for that FEC, AND it's configured to generate the same (non-NULL) label for all the FECs that share the same next hop and for which it's the egress, then it searches for an entry that indicates Label pop (means no outgoing label) with a next hop equal to the next hop for that FEC. (how you figure out the next hop depends on the type of FEC. like, if the FEC is an IP address prefix, the next hop comes from the IP forwarding table 🍓) found a match? no longer stale, associate the incoming label from that entry with the FEC, and advertise <incoming label, FEC> to neighbors via LDP. otherwise, normal LDP procedures.

next up: same egress vibe, non-NULL label, BUT it generates a unique label for each FEC 🦋 then the LSR needs to preserve the <incoming label, (outgoing label, next hop)> mapping AND the FEC tied to it. so it would search for an entry that indicates Label pop (means no outgoing label), and whose next hop equals the next hop for that FEC associated with the entry. (same note as before: the next hop for a FEC depends on its type, and for an IP address prefix it's from the IP forwarding table 💅) found? not stale anymore, associate the incoming label with the FEC, advertise <incoming label, FEC> to neighbors. otherwise, normal LDP procedures.

last one in this section: if an LSR is the egress for a particular FEC, and it's configured to generate a NULL label (either Explicit or Implicit) for that FEC, it just advertises that label, together with the FEC, to its neighbors via LDP. simple and iconic 💖

---

3.2. Alternative procedures for the restarting LSR 🌸

okay so in this section we're looking at an alternative to the procedures from Section 3.1, "Procedures for the restarting LSR" ✨

the procedures in this section assume that the restarting LSR has (at least) as many unallocated labels as allocated labels 🧚 the allocated ones are the MPLS forwarding state the LSR managed to preserve across the restart.

after an LSR restarts its control plane, it MUST check whether it was able to preserve its MPLS forwarding state from before the restart 😭 if it couldn't, then the LSR sets the Recovery Time to 0 in the FT Session TLV it sends to its neighbors.

if the forwarding state WAS preserved, the LSR starts its internal timer, called the MPLS Forwarding State Holding timer (the value of that timer SHOULD be configurable 🎀), and marks all the MPLS forwarding state entries as "stale". when the timer expires, all the entries still marked as stale SHOULD be deleted. the value of the Recovery Time advertised in the FT Session TLV is set to the (current) value of the timer at the point when the Initialization message carrying the FT Session TLV is sent. 🔥

we say that an LSR is in the process of restarting when the MPLS Forwarding State Holding timer hasn't expired yet. once the timer expires, we say the LSR completed its restart 🌈

while an LSR is in the process of restarting, it creates local label bindings by following the normal LDP procedures.

note that while it's restarting, the LSR may have not one, but TWO local label bindings for a given FEC 🙈 one retained from before the restart, and another created after the restart. once the LSR completes its restart, the former will be deleted. both of these bindings, though, would have the same outgoing label (and the same next hop) 💜

---

3.3. Restart of LDP communication with a neighbor LSR 🦋

okay so when an LSR detects that its LDP session with a neighbor went down, AND it knows that the neighbor is capable of preserving its MPLS forwarding state across the restart (which was indicated by the FT Session TLV in the Initialization message from that neighbor), the LSR retains the label-FEC bindings it got via that session (instead of tossing them) but marks them as "stale" 💅

after detecting that the LDP session went down, the LSR tries to re-establish LDP communication with the neighbor following the usual LDP procedures 🧚

the LSR keeps its stale label-FEC bindings for an amount of time set to the lesser of two things: the FT Reconnect Timeout (as advertised by the neighbor), and a local timer called the Neighbor Liveness Timer 🕰️ if within that time the LSR still hasn't established an LDP session with the neighbor, all the stale bindings SHOULD be deleted. the Neighbor Liveness Timer starts when the LSR detects that its LDP session went down, and the value of the Neighbor Liveness timer SHOULD be configurable ✨

if the LSR re-establishes an LDP session with the neighbor within the lesser of the FT Reconnect Timeout and the Neighbor Liveness Timer, AND it figures out the neighbor was NOT able to preserve its MPLS forwarding state, the LSR SHOULD immediately delete all the stale label-FEC bindings from that neighbor 💀 but if the neighbor WAS able to preserve its forwarding state (which shows up as a non-zero Recovery Time advertised by the neighbor), the LSR SHOULD further keep those stale label-FEC bindings for as long as the lesser of two things allows: the Recovery Time advertised by the neighbor, and a local configurable value called Maximum Recovery Time 🌸

the LSR SHOULD try to finish its label mapping exchange with the neighbor within 1/2 of the Recovery Time, as specified in the FT Session TLV received from the neighbor. no pressure, bestie 😭🫶

the LSR handles Label Mapping messages from the neighbor by following the normal LDP procedures, with a few twists 🍓 (a) it treats the stale entries in its Label Information Base (LIB) as if they had been received over the newly established session, (b) if the label-FEC binding in the message is the same as one in the LIB but that LIB entry is marked as stale, the LIB entry is no longer marked as stale, and (c) if, for the FEC in the label-FEC binding carried in the message, there's already a stale label-FEC binding in the LIB, and the label in the LIB binding is different from the label in the message, the LSR just updates the LIB entry with the new label. iconic behaviour 👑

an LSR, once it creates a <label, FEC> binding, SHOULD keep the value of that label for as long as the LSR has a route to the FEC in the binding 💖 if the route to the FEC disappears and then re-appears later, it may end up using a different label value, because when the route comes back, the LSR creates a new <label, FEC> binding.

to minimize the potential mis-routing caused by a label change when creating a new <label, FEC> binding, the LSR SHOULD pick up the least recently used label 🌈 and once an LSR releases a label, the LSR SHOULD NOT re-use that label for advertising a <label, FEC> binding to a neighbor that supports graceful restart for at least the sum of the FT Reconnect Timeout plus the Recovery Time, as advertised by that neighbor to the LSR. ngl that's a lot of timers, but they're literally keeping the network from getting confused 🔥🦋

4. Security Consideration ✨

Okay bestie, security time 🔒 The security stuff from the original LDP protocol [RFC3036] is still totally relevant, no cap.

On top of that, LSRs that use this mechanism have some extra denial-of-service risks to know about 😭

• An intruder can pretend to be an LDP peer so the TCP connection fails and has to reconnect. The intruder then sets the Recovery Time to 0 on that reconnection, which forces all labels received from the peer to be released. 💀
• An intruder could also sit in the middle of the traffic between LDP peers and override the Recovery Time so it is set to 0. Same result: all labels received from that peer get released. 🙈

Good news though: all of these can be countered by using an authentication scheme between LDP peers, like the MD5-based scheme described in [LDP] 🫶

Now for the label thing, which is very serious 🥺 As with LDP, there's a security issue if an LDP implementation keeps using labels after the session that first caused them to be used has expired. This can happen if the upstream LSR notices the session failed after the downstream LSR has already released and re-used the label. It's most obvious with the platform-wide label space, and it could end up misrouting data to places it wasn't meant to go. It's also possible that people deliberately exploit this to get services they aren't authorized for, or to deny services to others. Not it! 🚫

In this document, the session's validity can be extended by the Reconnect Timeout, and the session may be re-established during that window. Once the Reconnection Timeout has expired, the session must be considered to have failed, and the same security issue described above applies.

BUT here's the tricky part 👀 The downstream LSR might declare the session failed before its Reconnection Timeout runs out. That makes the window bigger in which the downstream LSR could re-allocate the label while the upstream LSR is still sending data using the old use of that label. To help with this, this document requires that labels not be re-used until at least the sum of the Reconnect Timeout plus the Recovery Time. Protect your labels, periodt 💅✨

5. Intellectual Property Considerations 🎀

This section is taken from Section 10.4 of [RFC2026].

Here's the legal-but-make-it-clear version 🧚 The IETF doesn't take a position on whether any intellectual property or other rights someone claims over the technology in this document are valid or in scope, or on whether licenses for those rights are available. It also doesn't say it has tried to find any such rights. Info on the IETF's process for rights in standards-track and standards-related documents is in BCP-11. Copies of claimed rights that were submitted for publication, any promised licenses, or the results of anyone trying to get a general license or permission to use those proprietary rights for implementers or users of this spec can be requested from the IETF Secretariat.

The IETF also invites any interested party to tell it about any copyrights, patents, patent applications, or other proprietary rights that might cover technology needed to practice this standard. Send that info to the IETF Executive Director.

The IETF has been told about intellectual property rights claimed for some or all of the specification in this document. For more details, check the online list of claimed rights.

6. Acknowledgments 🌸

Huge thanks to Loa Andersson, Chaitanya Kodeboyina, Ina Minei, Nischal Sheth, Enke Chen, and Adrian Farrel for their contributions to this document 🥰🫶

7. Normative References 📚

• [LDP] Andersson, L., Doolan, P., Feldman, N., Fredette, A. and B. Thomas, "Label Distribution Protocol", RFC 3036, January 2001.
• [FT-LDP] Farrel, A., "Fault Tolerance for the Label Distribution Protocol (LDP)", RFC 3479, February 2003.
• [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
• [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996.

8. Informative References 🔍

• [OSPF-RESTART] "Hitless OSPF Restart", Work in Progress.
• [ISIS-RESTART] "Restart signaling for ISIS", Work in Progress.
• [BGP-RESTART] "Graceful Restart Mechanism for BGP", Work in Progress.

9. Authors' Addresses 💌

Manoj Leelanivas
Juniper Networks
1194 N. Mathilda Ave
Sunnyvale, CA 94089

EMail: manoj@juniper.net

Yakov Rekhter
Juniper Networks
1194 N. Mathilda Ave
Sunnyvale, CA 94089

EMail: yakov@juniper.net

Rahul Aggarwal
Redback Networks
350 Holger Way
San Jose, CA 95134

EMail: rahul@redback.com

10. Full Copyright Statement 💜

Copyright (C) The Internet Society (2003). All Rights Reserved.

This document and translations of it may be copied and shared with others, and derivative works that comment on or otherwise explain it or help implement it may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind. The one catch is that the copyright notice and this paragraph must be included on all such copies and derivative works. This document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations. The exceptions are when it's needed for developing Internet standards, in which case the copyright procedures in the Internet Standards process must be followed, or when it's being translated into languages other than English.

The limited permissions above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information in it is provided on an "AS IS" basis, and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 🙌

Funding for the RFC Editor function is currently provided by the Internet Society. ✨