RFC 3386

Network Hierarchy and Multilayer Survivability

📅 published November 2002, from ietf

Network Hierarchy and Multilayer Survivability 🎀

Okay bestie, this is RFC 3386 🦋 and it's giving networking vibes! It's titled "Network Hierarchy and Multilayer Survivability" and it was written by two editors, W. Lai, Ed. from AT&T and D. McDysan, Ed. from WorldCom. It came out of the Network Working Group in November 2002 and it's categorized as Informational, so it's info, not a standard 💅 The header doesn't list anything it obsoletes or updates, so it stands on its own ✨

Status of this Memo 📜

This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.

Copyright Notice ©️

Abstract 🌸

This doc proposes the near-term, practical requirements for network survivability and hierarchy in current service provider environments. Basically, it's a realistic wishlist for today's networks 🫶

Conventions used in this document 📖

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [2]. So when you see those words in capitals, they carry the BCP 14 meaning, no vibes-based guessing 🙈

Table of Contents 🗂️

   1. Introduction..............................................2
   2. Terminology and Concepts..................................5
   2.1 Hierarchy................................................6
   2.1.1 Vertical Hierarchy.....................................5
   2.1.2 Horizontal Hierarchy...................................6
   2.2 Survivability Terminology................................6
   2.2.1 Survivability..........................................7
   2.2.2 Generic Operations.....................................7
   2.2.3 Survivability Techniques...............................8
   2.2.4 Survivability Performance..............................9
   2.3 Survivability Mechanisms: Comparison....................10
   3. Survivability............................................11
   3.1 Scope...................................................11
   3.2 Required initial set of survivability mechanisms........12
   3.2.1 1:1 Path Protection with Pre-Established Capacity.....12
   3.2.2 1:1 Path Protection with Pre-Planned Capacity.........13
   3.2.3 Local Restoration.....................................13
   3.2.4 Path Restoration......................................14
   3.3 Applications Supported..................................14
   3.4 Timing Bounds for Survivability Mechanisms..............15
   3.5 Coordination Among Layers...............................16
   3.6 Evolution Toward IP Over Optical........................17
   4. Hierarchy Requirements...................................17
   4.1 Historical Context......................................17
   4.2 Applications for Horizontal Hierarchy...................18
   4.3 Horizontal Hierarchy Requirements.......................19
   5. Survivability and Hierarchy..............................19
   6. Security Considerations..................................20
   7. References...............................................21
   8. Acknowledgments..........................................22
   9. Contributing Authors.....................................22
   Appendix A: Questions used to help develop requirements.....23
   Editors' Addresses..........................................26
   Full Copyright Statement....................................27

1. Introduction ✨

This doc is the result of the Network Hierarchy and Survivability Techniques Design Team, which was set up inside the Traffic Engineering Working Group. The team collected and documented current and near-term requirements for survivability and hierarchy in service provider environments. For clarity, they also included an expanded set of definitions. The team figured there seems to be a need to define a small set of interoperable survivability approaches in packet and non-packet networks. Suggested approaches include path-based ones, plus one that repairs connections close to the network fault. They mostly operate at a single network layer.

For hierarchy, there didn't seem to be a driving near-term need to work on "vertical hierarchy," which means communication between network layers like Time Division Multiplexed (TDM)/optical and Multi-Protocol Label Switching (MPLS). Instead of directly exchanging signaling and routing between vertical layers, some looser coordination, like specifying hold-off timers, is the nearer-term need 🧚

For "horizontal hierarchy" in data networks, there are several pressing needs 🔥 The requirement is to be able to set up many Label Switched Paths (LSPs) in a service provider network with hierarchical Interior Gateway Protocol (IGP). This is needed to support layer 2 and layer 3 Virtual Private Network (VPN) services that require edge-to-edge signaling across a core network.

This doc proposes near-term, practical requirements for network survivability and hierarchy in current service provider environments. With feedback from the working group, the goal is to help focus the work happening in TEWG (Traffic Engineering Working Group), CCAMP (Common Control and Measurement Plane Working Group), and other working groups. A main goal is to provide some expedience for required functionality in multi-vendor service provider networks. The initial focus is mainly on intra-domain operations. However, to keep end-to-end service consistent in a multi-provider environment, rules governing how survivability mechanisms operate at domain boundaries must also be specified. While such issues are raised and discussed where appropriate, they won't be treated in depth in the initial release of this doc 🌈

The doc first builds a set of definitions to be used later in this doc and potentially in other docs. Then it covers the requirements and issues around service restoration and hierarchy, and finally gives a short discussion of survivability in a hierarchical context.

Here's a summary of the findings:

A. Survivability Requirements

need to define a small set of interoperable survivability approaches in packet and non-packet networks
suggested survivability mechanisms include
- 1:1 path protection with pre-established backup capacity (non-shared)
- 1:1 path protection with pre-planned backup capacity (shared)
- local restoration with repairs in proximity to the network fault
- path restoration through source-based rerouting
timing bounds for service restoration to support voice call cutoff (140 msec to 2 sec), protocol timer requirements in premium data services, and mission critical applications
use of restoration priority for service differentiation

B. Hierarchy Requirements

B.1. Horizontally Oriented Hierarchy (Intra-Domain)

ability to set up many LSPs in a service provider network with hierarchical IGP, for the support of layer 2 and layer 3 VPN services
requirements for multi-area traffic engineering need to be developed to provide guidance for any necessary protocol extensions

B.2. Vertically Oriented Hierarchy

The following functionality for survivability is common on most routing equipment today. 🖥️

near-term need is some loose form of coordination and communication based on the use of nested hold-off timers, instead of direct exchange of signaling and routing between vertical layers
means for an upper layer to immediately begin recovery actions in the event that a lower layer is not configured to perform recovery

C. Survivability Requirements in Horizontal Hierarchy

protection of end-to-end connection is based on a concatenated set of connections, each protected within their area
mechanisms for connection routing may include (1) a network element that participates on both sides of a boundary (e.g., OSPF ABR) - note that this is a common point of failure; (2) a route server
need for inter-area signaling of survivability information (1) to enable a "least common denominator" survivability mechanism at the boundary; (2) to convey the success or failure of the service restoration action; e.g., if a part of a "connection" is down on one side of a boundary, there is no need for the other side to recover from failures 💀

2. Terminology and Concepts 🧠

2.1 Hierarchy

Hierarchy is a technique used to build scalable complex systems. It's based on an abstraction, at each level, of what is most significant from the details and internal structures of the levels further away. This approach uses a general property of all hierarchical systems made of related subsystems: interactions between subsystems decrease as the level of communication between those subsystems decreases. Main character energy, honestly 👑

Network hierarchy is an abstraction of part of a network's topology, routing and signaling mechanisms. Abstraction may be used as a way to build large networks, or as a technique for enforcing administrative, topological, or geographic boundaries. For example, network hierarchy might be used to separate the metropolitan and long-haul regions of a network, or to separate the regional and backbone sections of a network, or to interconnect service provider networks (with BGP, which reduces a network to an Autonomous System).

In this document, network hierarchy is considered from two perspectives:

(1) Vertically oriented: between two network technology layers.

(2) Horizontally oriented: between two areas or administrative subdivisions within the same network technology layer.

2.1.1 Vertical Hierarchy

Vertical hierarchy is the abstraction, or reduction in information, that would be of benefit when communicating information across network technology layers, like propagating information between optical and router networks. 🦋

In the vertical hierarchy, the total network functions are partitioned into a series of functional or technological layers, with clear logical, and maybe even physical, separation between adjacent layers. Survivability mechanisms either currently exist or are being developed at multiple layers in networks [3]. The optical layer is now becoming capable of providing dynamic ring and mesh restoration functionality, in addition to traditional 1+1 or 1:1 protection. The Synchronous Digital Hierarchy (SDH)/Synchronous Optical NETwork (SONET) layer provides survivability capability with automatic protection switching (APS), as well as self-healing ring and mesh restoration architectures. Similar functionality has been defined in the Asynchronous Transfer Mode (ATM) Layer, with work ongoing to also provide such functionality using MPLS [4]. At the IP layer, rerouting is used to restore service continuity following link and node outages. Rerouting at the IP layer, however, occurs after a period of routing convergence, which may require a few seconds to several minutes to complete [5] 🫠

2.1.2 Horizontal Hierarchy ✨

Okay bestie, horizontal hierarchy is the trick that lets a network at one technology layer, like a packet network, scale 📈🌸 Real examples are BGP confederations, separate Autonomous Systems, and multi-area OSPF.

The big idea is that a large network gets split into multiple smaller sub-networks that don't overlap 🧩 You can slice it up based on topology, network function, administrative policy, or service domain demarcation. Two networks at the same hierarchical level (think two Autonomous Systems in BGP) can share a peer relation through some loosely coupled connection 🫶 Meanwhile, multi-area OSPF gets its abstraction by aggregating routing information, and that works because the network has been partitioned hierarchically. It's giving organized chaos, but make it efficient 💅

2.2 Survivability Terminology 🦋

Here's the alphabetical list of terms defined in this section. Lowkey a lot, but you'll get the hang of it 💜

backup entity, same as protection entity (section 2.2.2)
extra traffic (section 2.2.2)
non-revertive mode (section 2.2.2)
normalization (section 2.2.2)
preemptable traffic, same as extra traffic (section 2.2.2)
preemption priority (section 2.2.4)
protection (section 2.2.3)
protection entity (section 2.2.2)
protection switching (section 2.2.3)
protection switch time (section 2.2.4)
recovery (section 2.2.2)
recovery by rerouting, same as restoration (section 2.2.3)
recovery entity, same as protection entity (section 2.2.2)
restoration (section 2.2.3)
restoration priority (section 2.2.4)
restoration time (section 2.2.4)
revertive mode (section 2.2.2)
shared risk group (SRG) (section 2.2.2)
survivability (section 2.2.1)
working entity (section 2.2.2)

2.2.1 Survivability 🎀

Survivability is a network's ability to keep service going even when there are faults inside the network [6] 🔥 Protection and restoration are the main mechanisms for this. They can be done per link, per path, or across the whole network, all to cut down service disruption without going broke 💸 How survivable a network is depends on whether it can handle single failures, multiple failures, and equipment failures.

2.2.2 Generic Operations 🌈

Quick heads-up: this document doesn't walk through the exact order of events for how failures get monitored, detected, and handled. If you want that deep dive, check out [4] 📚 The repair process after a failure is also out of scope here, so we're skipping it 🙈

A working entity is what carries traffic during normal operation. Depending on the context, it could be a channel or transmission link at the physical layer, a Label Switched Path (LSP) in MPLS, or a logical bundle of one or more LSPs 🧚

A protection entity (also called a backup entity or recovery entity) is what carries protected traffic in recovery operation mode, which is when the working entity is in error or has failed 💖

Extra traffic (aka preemptable traffic) is whatever rides the protection entity while the working entity is active ✨ It isn't protected. So if the protection entity has to step in to protect the traffic on the working entity, the extra traffic gets preempted 💀

A shared risk group (SRG) is a set of network elements that can all get hit by the same fault or fault type 🍓 For example, a shared risk link group (SRLG) is every link on fibers routed through the same physical conduit in a fiber-span network. It goes beyond shared conduit too, with things like shared fiber cable, shared right of way, shared optical ring, or a shared office without power sharing 🔌 The span of an SRG (like how long the sharing goes for compromised outside plant) needs to be looked at for each fault basis. The SRG idea can also stretch into a "risk domain," plus its capabilities and summarization for traffic engineering purposes 🌸 Check [7] for more tea on this one.

Normalization is the sequence of events and actions a network takes to get back to its preferred state once a failure is repaired ✨ That can include switching or rerouting affected traffic back onto the repaired working entities or onto new routes 🛤️ Revertive mode is when traffic gets automatically moved back to a repaired working entity (also called switch back) 🔁

Recovery is the sequence of events and actions a network runs after spotting a failure. The goal is to keep existing services at the performance level they need (say, per service level agreements) and to allow normalization 👑 It kicks off with a failure notification, then splits into two parallel processes: (1) a repair process, which isolates the fault and fixes the broken parts, and (2) a reconfiguration process that uses survivability mechanisms to keep service continuity going 🫶 With protection, reconfiguration means switching the affected traffic from a working entity to a protection entity. With restoration, it means path selection and rerouting for the affected traffic 🦋

Revertive mode is the procedure where a revertive action happens (that's a switch back from the protection entity to the working entity) once the failed working entity has been repaired 🌈 In non-revertive mode, that action doesn't happen. To keep service interruption low, a switch-back in revertive mode should be done when it'll hit the affected traffic the least, or with the make-before-break concept 🎀

Non-revertive mode is for cases where there's no preferred path, or where it's smarter to avoid extra disruption from a revertive switch 💅 Switching back to the original working path isn't wanted or possible, since that original path might not even exist anymore after a fault on it 💀

2.2.3 Survivability Techniques 🧚

Protection, also called protection switching, is a survivability technique based on predetermined failure recovery. As soon as the working entity is set up, a protection entity gets set up too ✨ There are a few architectures for this: 1+1, 1:1, 1:n, and m:n. In SDH/SONET, these get called Automatic Protection Switching (APS) 🔥

In the 1+1 architecture, every working entity gets its own dedicated protection entity 💜 It uses a dual-feed mechanism, where the working entity is permanently bridged onto the protection entity at the source of the protected domain 🌸 In normal operation, identical traffic goes out on both the working and protection entities at the same time. At the sink end of the protected domain, both feeds get watched for alarms and maintenance signals 👀 The choice between the two is made using some predetermined criteria, like transmission performance requirements or defect indication 🎯

In the 1:1 architecture, every working entity also gets a dedicated protection entity 🍓 The protected traffic normally goes over the working entity. When that working entity fails, the protected traffic gets switched to the protection entity. Both ends of the protected domain have to signal that they've detected the fault and kick off the switchover 🤝

In the 1:n architecture, one dedicated protection entity is shared by n working entities 🌈 Heads up: not all of the affected traffic may end up protected in this setup 🙈

The m:n architecture is a generalization of 1:n. Typically m <= n, meaning m dedicated protection entities are shared by n working entities 👑

Restoration, also called recovery by rerouting [4], is a survivability technique that sets up new paths or path segments on demand to restore affected traffic after a fault ⚡ The resources on those alternate paths are the currently unassigned (unreserved) resources in the same layer 🛤️ If there aren't spare resources to carry the higher-priority protected traffic, preempting extra traffic may also be used 💀 Once a fault is detected on the working path, the recovery path can be picked based on preplanned configurations, network routing policies, or the current network status (like the topology and fault information) 🧭 Signaling is what sets up the new paths that bypass the fault. So basically, restoration is a path selection process followed by rerouting the affected traffic from the working entity to the recovery entity 🦋

2.2.4 Survivability Performance 💖

Protection switch time is the time from when a network fault happens until the protection-switching operations are complete ⏱️ It covers the detection time needed to kick off the protection switch, any hold-off time that lets protection schemes interwork, and the switch completion time 🎀

Restoration time is the time from a network fault to the moment the affected traffic is either completely restored, or spare resources run out, and/or there's no more extra traffic left that can be preempted to make room 🫠

Restoration priority is a way of giving preference to protecting higher-priority traffic ahead of lower-priority traffic 👑 It helps decide the order in which traffic gets restored after a failure 🌸 The point is to differentiate service restoration time, and to control who gets access to the available spare capacity for different classes of traffic 💅

Preemption priority is a way of figuring out which traffic can be disconnected if not all of the traffic with a higher restoration priority gets restored after a failure 🥺💜

2.3 Survivability Mechanisms: Comparison ✨

okay so in a survivable network, spare capacity and diversity must be built in from day one so the network can self-heal whenever something breaks 🩹 the common move is to pair each working entity with a protection entity, and that protection entity has either dedicated resources or shared ones that are pre-reserved or reserved-on-demand 💅 depending on how you set up that protection entity, you get different survivability approaches, and they can be sorted into groups 🗂️ protection techniques usually rely on a dedicated protection entity that's already set up before anything fails 🛡️ restoration techniques are the opposite vibe. they mostly lean on spare capacity sitting around in the network 🌊 so the trade-off is pretty clear: protection gives you fast recovery and better availability, while restoration gives you better resource utilization 🫶

a 1+1 protection architecture is rather expensive, ngl 💸 because you need duplicate resources for both the working and protection entities. it's generally saved for specific services that need a very high availability 👑

a 1:1 architecture is inherently slower at recovering from failure than 1+1 🐢 both ends of the protection domain have to communicate to do the switch-over. the perk is that the protection entity can optionally carry low-priority extra traffic during normal operation, but only if traffic preemption is allowed 🦋 packet networks can pre-establish a protection path for later use with pre-planned capacity that is not pre-reserved 🧚 which means if no packets are sent onto a protection path, no bandwidth gets eaten at all. transmission networks like optical or TDM are not like that, because path establishment and resource reservation can't be decoupled there 🙈

in the 1:n protection architecture, traffic normally runs on the working entities 🛤️ if several working entities fail at the same time, only one of them can be restored by the common protection entity 😭 this contention can be sorted out by giving each working entity a different preemptive priority 🥇 and just like the 1:1 case, the protection entity can optionally carry preemptable traffic during normal operation 🌸

the m:n architecture can boost system availability with small cost increases, which is iconic 🤩 but it has rarely been implemented or standardized, so it's kind of a hidden gem 💎

compared to protection, restoration is generally more frugal 🌱 nothing gets committed until after the fault happens and its location is known. the downside is that restoration is inherently slower, since more has to happen after the fault is detected 🐌 the time it takes to dynamically pick and set up alternate paths can change depending on how much traffic and how many connections need restoring, plus the network topology, the technology in use, and the type and severity of the fault 🌪️ so restoration time tends to be way more variable than a protection switch with pre-selected protection entities 📉 that's why, when using restoration, it's essential to use restoration priority so service objectives get met cost-effectively 💜

once the network routing algorithms have converged after a fault, it may be preferable in some cases to reoptimize the network by rerouting based on the current network state and network policies 🔄✨

3. Survivability 👑

3.1 Scope 🔭

okay so interoperable approaches to network survivability were agreed to be an immediate requirement in packet networks and in SDH/SONET framed TDM networks 💖 not as pressing at this point were techniques covering all-optical networks (like where framing is unknown), because controlling those in a multi-vendor environment seemed to have some other hurdles to clear first 🧗‍♀️ also not of immediate interest were approaches that coordinate or explicitly communicate survivability mechanisms across network layers (like from a TDM or optical network to or from an IP network) 🙅‍♀️ BUT a capability should be provided for a network operator to perform fault notification and control survivability mechanisms among different layers 🫶 this may need some corresponding OAM functionality. however, those issues, and the ones related to OAM, are currently outside the scope of this document 🚧 (for proposed MPLS OAM requirements, see [8, 9]) 🔗

the initial scope is to address only "backhoe failures" in the inter-office connections of a service provider network 🚜 a link connection in the router layer is typically made up of multiple spans in the lower layers, so the network failures that trigger a recovery include link/span failures 🧩 however, linecard and node failures may not need to be treated any differently from their respective link/span failures, since a router failure can be represented as a set of simultaneous link failures 🤓

depending on the actual network configuration, a drop-side interface (like between a customer and an access router, or between a router and an optical cross-connect) may count as either inter-domain or inter-layer 🎀 another inter-domain scenario is using intra-office links to interconnect a metro network and a core network when both are run by the same service provider 🌐 failures at those interfaces may be protected by the mechanisms of this section in a similar way 💅

other, more complex failures are not in scope for the survivability mechanisms discussed here 🙈 that includes systematic control-plane failure, configuration error, or breach of security 🔐 network impairment like congestion that lowers throughput is also not covered 🚫

3.2 Required initial set of survivability mechanisms 🎀

3.2.1 1:1 Path Protection with Pre-Established Capacity 🛡️

in this protection mode, the head end of a working connection sets up a protection connection to the destination 🧭 there should be the ability to keep relative restoration priorities between working and protection connections, as well as between different classes of protection connections 🏷️

in normal operation, traffic is only sent on the working connection 🛤️ though the ability to signal that traffic will go on both connections (1+1 Path for signaling purposes) would be super valuable in non-packet networks 🤩 some distinction between working and protection connections is likely, either through explicit objects or, preferably, through implicit methods like general classes or priorities 🏷️ head ends need the ability to create connections that are as failure disjoint from each other as possible 🪓 this requires SRG information that can generally be assigned to either nodes or links and propagated through the control or management plane 📡 in this mechanism, capacity in the protection connection is pre-established, but it should be capable of carrying preemptable extra traffic in non-packet networks 🦋 when protection capacity gets called into service during recovery, there should be the ability to promote the protection connection to working status (for non-revertive mode operation) with some form of make-before-break capability 🪄

3.2.2 1:1 Path Protection with Pre-Planned Capacity 🗓️

this is similar to the 1:1 protection with pre-established capacity above, since the protection connection is also pre-signaled 🤝 the difference is in how protection capacity gets assigned 🎟️ with pre-planned capacity, the mechanism supports protection capacity being shared, or "double-booked" 👯 operators need the ability to provision different amounts of protection capacity based on expected failure modes and service level agreements 📋 so an operator might want enough restoration capacity to handle a single failure affecting all connections in an SRG, or might want to provision less or more than that 🧮 mechanisms should be provided so restoration capacity on each link can be shared by SRG-disjoint failures 🤝 in a sense this is 1:1 from a path perspective, but the protection capacity in the network (on a link by link basis) is shared in a 1:n fashion, e.g., see the proposals in [10, 11] 🧠 if capacity is planned but not allocated, some form of signaling could be required before traffic may be sent on protection connections, especially in TDM networks 📶

the use of this approach improves network resource utilization, but it may need more careful planning 📝 so initial deployment might be based on 1:1 path protection with pre-established capacity plus the local restoration mechanism described next 🪜

3.2.3 Local Restoration 🔧

due to the time impact of signal propagation, dynamic recovery of an entire path may not meet the service requirements of some networks ⏱️ the solution is to restore connectivity of the link or span in immediate proximity to the fault, e.g., see the proposals in [12, 13] 📍 at a minimum, this approach should be able to protect against connectivity-type SRGs, though protecting against node-based SRGs might be worthwhile 🌟 this approach also applies to supporting restoration on the inter-domain and inter-layer interconnection scenarios using intra-office links, as described in the Scope Section 🔁

head end systems must have some control over whether their connections are candidates for local restoration or excluded from it 🎛️ for example, best-effort and preemptable traffic may be excluded from local restoration, so they only get restored if there is bandwidth available 🫧 this type of control may require the definition of an object in signaling 📦

since local restoration may be suboptimal, a means for head end systems to later perform path-level re-grooming must be supported for this approach 🧹

3.2.4 Path Restoration 🛣️

in this approach, connections impacted by a fault get rerouted by the originating network element once it's notified of the connection failure 📣 this source-based approach is efficient for network resources, but it typically takes longer to get restoration done 🐢 it doesn't involve any new mechanisms 🙌 it's really just a mention of another common approach to protecting against faults in a network, nothing more 🤷‍♀️

3.3 Applications Supported 📱

okay, bestie, so with service continuity under failure as the goal, a network is "survivable" if, when a failure hits, connectivity gets interrupted for a "brief" period and then comes back before the network failure ends 💫 how long that interruption can be depends on the application 🫠 here are some typical applications and the considerations that drive what an acceptable protection switch time or restoration time looks like:

Best-effort data: recovering network connectivity by rerouting at the IP layer would be sufficient 🌊
Premium data service: needs to meet TCP timeout or application protocol timer requirements ⏲️
Voice: call cutoff is in the range of 140 msec to 2 sec (that's the time a person waits after the speech path is interrupted before hanging up, or the time a telephone switch will disconnect a call) 📞😬
Other real-time service (e.g., streaming, fax) where an interruption would cause the session to terminate 🎬
Mission-critical applications that cannot tolerate even brief interruptions, for example, real-time financial transactions 💸🚨

3.4. Timing Bounds for Survivability Mechanisms

okay so the plan was to look at a whole spectrum of survivability mechanisms, each protecting traffic with different levels of survivability and different protection/restoration speeds 🦋. Then the team picked a few general points that cover that spectrum 💅. The goal is requirements, not a pile of detailed proposals, so an operator still gets some (limited) flexibility when engineering multi-vendor networks 🫶. The requirements of different applications from the last subsection got discussed, but ngl nobody on the team would really swear the timing bounds below scientifically match any specific app's needs 😭. A few assumptions:

Approaches in which protection switch without propagation of information are likely to be faster than those that do require some form of fault notification to some or all elements in a network.
Approaches that require some form of signaling after a fault will also likely suffer some timing impact.

Here are the proposed timing bounds for each survivability mechanism (all bounds are exclusive of signal propagation) ⏱️:

1:1 path protection with pre-established capacity:  100-500 ms
1:1 path protection with pre-planned capacity:      100-750 ms
Local restoration:                                  50 ms
Path restoration:                                   1-5 seconds

To make sure the service requirements for different applications can be met within those timing bounds, restoration priority must be implemented 👑. It decides the order in which connections get restored, which helps minimize service restoration time and gives access to available spare capacity on the best paths 🌈. For example, mission critical applications may require high restoration priority 🚨. At the fiber layer, instead of specific applications, it may be possible that priority be given to certain classifications of customers, with their traffic types enclosed within the customer aggregate 🧚.

Preemption priority is the one that only comes into play when not all connections can be restored. In that case, connections with lower preemption priority should be released 🙈. Depending on a service provider's strategy in provisioning network resources for backup, preemption may or may not be needed in the network 🤷‍♀️.

3.5. Coordination Among Layers

okay this is giving layers and how they talk to each other 🍓. A common design goal for networks with multiple technological layers is to provide the desired level of service in the most cost-effective manner 💖. Multilayer survivability may allow the optimization of spare resources, by sharing spare capacity across different layers and improving resource utilization, though further investigations are needed 🔍. Coordination during recovery among different network layers (e.g., IP, SDH/SONET, optical layer) might necessitate development of vertical hierarchy 🪜. The benefits of providing survivability mechanisms at multiple layers, and the optimization of the overall approach, must be weighed with the associated cost and service impacts ⚖️.

A default coordination mechanism for inter-layer interaction could be the use of nested timers and current SDH/SONET fault monitoring, as has been done traditionally for backward compatibility 🕰️. So when lower-layer recovery takes a longer time period than higher-layer recovery, a hold-off timer is used to avoid contention between the different single-layer survivability schemes 🛑. In other words, multilayer interaction is handled by having successively higher multiplexing levels operate at a protection/restoration time scale greater than the next lowest layer 📶. This can impact the overall time to recover service 😬. For example, if SDH/SONET protection switching is used, MPLS recovery timers must wait until SDH/SONET has had time to switch ⏳. Setting such timers is a tradeoff between rapid recovery and creating a race condition where multiple layers respond to the same fault, potentially allocating resources in an inefficient manner 🌀.

In other configurations where the lower layer does not have a restoration capability or is not expected to protect (say, an unprotected SDH/SONET linear circuit), there must be a mechanism for the lower layer to trigger the higher layer to take recovery actions immediately 🚀. This difference in network configuration means that implementations must allow for adjustment of hold-off timer values and/or a means for a lower layer to immediately indicate to a higher layer that a fault has occurred, so the higher layer can take restoration or protection actions 💜.

Furthermore, faults at higher layers should not trigger restoration or protection actions at lower layers [3, 4] 🙅‍♀️.

It was felt that the current approach to coordinating survivability did not have significant operational shortfalls 🎀. These approaches include protecting traffic solely at one layer (e.g., at the IP layer over linear WDM, or at the SDH/SONET layer) 🌸. Where survivability mechanisms might be deployed at several layers, such as when a routed network rides a SDH/SONET protected network, it was felt that current coordination approaches were sufficient in many cases 👍. One exception is the hold-off of MPLS recovery until the completion of SDH/SONET protection switching, as described above. This limits the recovery time of fast MPLS restoration 😤. Also, by design, the operations and mechanisms within a given layer tend to be invisible to other layers 🫥.

3.6. Evolution Toward IP Over Optical

so the big picture here is where things are heading 🔮. As more pressing requirements for survivability and horizontal hierarchy for edge-to-edge signaling get met with technical proposals, it is believed that the benefits of merging (in some manner) the control planes of multiple layers will be outlined 🧠. When those benefits are self-evident, it would then seem to be the right time to review whether vertical hierarchy mechanisms are needed, and what the requirements might be 📋. For example, a future requirement might be to provide a better match between the recovery requirements of IP networks and the recovery capability of optical transport 💡. One such proposal is described in [14] ✨.

4. Hierarchy Requirements ✨

Efforts in the area of network hierarchy should focus on mechanisms that would allow more scalable edge-to-edge signaling, or signaling across networks with existing network hierarchy (such as multi-area OSPF) 🏗️. This appears to be a more urgent need than mechanisms that might be needed to interconnect networks at different layers 🚨.

4.1. Historical Context

okay story time 📖. One reason for horizontal hierarchy is functionality (e.g., metro versus backbone) 🏙️. Geographic "islands" or partitions reduce the need for interoperability and make administration and operations less complex 🏝️. Using a simpler, more interoperable, survivability scheme at metro/backbone boundaries is natural for many provider network architectures 🧩. In transmission networks, creating geographic islands of different vendor equipment has been done for a long time, because multi-vendor interoperability has been difficult to achieve 😅. Traditionally, providers have to coordinate the equipment on either end of a "connection," and making this interoperable reduces complexity 🔧. A provider should be able to concatenate survivability mechanisms in order to provide a "protected link" to the next higher level 🔗. Think of SDH/SONET rings connecting to TDM DXCs with 1+1 line-layer protection between the ADM and the DXC port 💍. The TDM connection, e.g., a DS3, is protected, but usually all equipment on each SDH/SONET ring is from a single vendor 🏷️. The DXC cross connections are controlled by the provider, and the ports are physically protected, resulting in a highly available design 🛡️. Thus, concatenation of survivability approaches can be used to cascade across a horizontal hierarchy ⛓️. While not perfect, it is workable in the near to mid-term until multi-vendor interoperability is achieved 🌤️.

While the problems associated with multi-vendor interoperability may necessitate horizontal hierarchy as a practical matter in the near to mid-term (at least this has been the case in TDM networks), there should not be a technical reason for it in the standards developed by the IETF for core networks, or even most access networks 🌐. Establishing interoperability of survivability mechanisms between multi-vendor equipment in core IP networks is urgently required to enable adoption of IP as a viable core transport technology, and to facilitate the traffic engineering of future multi-service IP networks [3] 🚀💅.

Some of the largest service provider networks currently run a single area/level IGP 🐢. Some service providers, as well as many large enterprise networks, run multi-area Open Shortest Path First (OSPF) to gain increases in scalability 📈. Often, this was from an original design, so it is difficult to say if the network truly required the hierarchy to reach its current size 🤔.

Some proposals on improved mechanisms to address network hierarchy have been suggested [15, 16, 17, 18, 19] 💭. This document aims to provide the concrete requirements so that these and other proposals can first aim to meet some limited objectives 🎯🫶.

4.2 Applications for Horizontal Hierarchy ✨

okay so the big reason we even care about horizontal hierarchy is signaling 💅 specifically edge-to-edge VPNs, which might be running across traffic-engineered data networks 🛣️. There are a bunch of ways to do layer 2 and layer 3 VPNs, and different emerging protocols are tackling them right now. You've got provider-provisioned VPNs (e.g., virtual routers) and the Pseudo Wire Edge-to-Edge Emulation (PWE3) efforts, which run on MPLS and/or IP tunnels. These might or might not need explicit edge-to-edge signaling 🤷‍♀️, but everyone kinda assumes that to hit SLAs you'll need some form of edge-to-edge signaling. it's giving needed, iykyk 🎀

now the scalability part is about dodging the O(N^2) properties of edge-to-edge signaling, where N is the number of edges 😭. but honestly the real problem isn't the raw amount of signaling. like, the sad scenario is O(N^2) meshes where there's a "connection" between every single edge pair 💀. even if setting up and maintaining those connections is doable in a big network, it could mess with the core survivability mechanisms 🛡️, and then protection and restoration times would grow with N^2. not cute at all. some value of N is kind of unavoidable, but finding ways to shrink N (for example, pulling things in from the edge to aggregation points) could be worth it 🌸

so most service providers feel like O(N^2) meshes aren't needed for VPNs, and that the number of tunnels for VPNs should fit inside what current protocols and implementations can handle 🧚. and that might be true, because right now there's a lack of ability to signal MPLS tunnels from edge to edge across an IGP hierarchy, like OSPF areas 🙈. this might mean we need signaling standards that support dynamic establishment and potentially restoration of LSPs across a 2-level IGP hierarchy. big ask, but iconic if it happens 👑

for routing scalability, especially in data applications, the big worry is how much processing and state each network element has to handle 🧠. if some nodes can't communicate and process the state of every other node, it might be better to limit the information they get 🚫. there's one school of thought that says the amount of information a horizontal barrier holds should be significant, and that the impacts this has on route selection optimality and on the ability to provide global survivability are accepted tradeoffs ⚖️. main character tradeoffs fr 🎭

4.3 Horizontal Hierarchy Requirements 🌈

mechanisms are needed so you can do edge-to-edge signaling of connections through a network 💖. one network scenario is medium to large networks that already have hierarchical interior routing, like multi-area OSPF or multi-level Intermediate System to Intermediate System (IS-IS) 🗺️. the main use case here is edge-to-edge signaling, which is thought to be necessary for keeping the SLAs of the layer 2 and layer 3 VPNs running across the network 💅. another possible context is edge-to-edge signaling in TDM SDH/SONET networks with IP control, where metro and core networks might also sit in a hierarchical interior routing domain 🍓

to support edge-to-edge signaling in those scenarios inside existing horizontal hierarchies, current traffic engineering (TE) methods [20, 6] might need an upgrade 🔧. requirements for multi-area TE need to be written so there's guidance for any protocol extensions that are necessary 📝 (we're literally building the blueprint, bestie 🫶)

5. Survivability and Hierarchy 🦋

when horizontal hierarchy exists in a network technology layer, there's a real question about how survivability can be provided along a connection that crosses those hierarchical boundaries 🤔

when designing protocols to meet hierarchy requirements, the approach to consider is that boundaries are either clean or of minimal value ✨. but the idea of network elements that participate on both sides of a boundary might be worth thinking about (e.g., OSPF ABRs) 👀. that lets devices on either side take an intra-area approach within their own region of knowledge 🏠, and lets the ABR do this in both areas and splice the two protected connections together at a common point. the catch is that this common point is now a common point of failure, so yeah, not ideal 😬. if the limitations of this approach start showing up in operational settings, then maybe it's time to start thinking about route-servers and signaling propagated directives 🚀

one initial approach might be to signal through a common border router and consider the service protected, since it's made of a concatenated set of connections that are each protected within their area 🔗. another approach might be a least common denominator mechanism at the boundary, e.g., 1+1 port protection 🛡️. there should also be some standardized means for a survivability scheme on one side of such a boundary to communicate with the scheme on the other side about whether the recovery action succeeded or failed 📡. for example, if part of a "connection" is down on one side of such a boundary, the other side doesn't need to recover from failures, so no double work 🙅‍♀️

in summary, at this time, approaches like the ones above that let survivability schemes be concatenated across hierarchical boundaries seem sufficient 💜

6. Security Considerations 🔐

the set of SRGs defined for a network under common administrative control, and the way those SRGs get assigned to nodes and links inside that administrative control, is sensitive information and needs to be protected 🙊. an SRG is basically an acknowledgement that the nodes and links in it are vulnerable to a common threat ⚠️. an adversary who has access to what's in an SRG could use that info to design an attack, figure out how much damage it would do, and therefore maximize the effect of the attack 💀. so yes, this is serious, bestie 😭

the label used to refer to a particular SRG has to allow for an encoding where sensitive stuff like physical location, function, purpose, customer, fault type, etc. is not easily discernable by unauthorized users 🕵️‍♀️

SRG information that gets propagated through the control and management plane should allow for an encryption mechanism 🔒. one example of an approach would be to use IPSEC [21] on all packets carrying SRG information 🛡️✨

7. References 📚

[1] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996.
[2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[3] K. Owens, V. Sharma, and M. Oommen, "Network Survivability Considerations for Traffic Engineered IP Networks", Work in Progress.
[4] V. Sharma, B. Crane, S. Makam, K. Owens, C. Huang, F. Hellstrand, J. Weil, L. Andersson, B. Jamoussi, B. Cain, S. Civanlar, and A. Chiu, "Framework for MPLS-based Recovery", Work in Progress.
[5] M. Thorup, "Fortifying OSPF/ISIS Against Link Failure", http://www.research.att.com/~mthorup/PAPERS/lf_ospf.ps
[6] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I. and X. Xiao, "Overview and Principles of Internet Traffic Engineering", RFC 3272, May 2002.
[7] S. Dharanikota, R. Jain, D. Papadimitriou, R. Hartani, G. Bernstein, V. Sharma, C. Brownmiller, Y. Xue, and J. Strand, "Inter-domain routing with Shared Risk Groups", Work in Progress.
[8] N. Harrison, P. Willis, S. Davari, E. Cuevas, B. Mack-Crane, E. Franze, H. Ohta, T. So, S. Goldfless, and F. Chen, "Requirements for OAM in MPLS Networks," Work in Progress.
[9] D. Allan and M. Azad, "A Framework for MPLS User Plane OAM," Work in Progress.
[10] S. Kini, M. Kodialam, T.V. Lakshman, S. Sengupta, and C. Villamizar, "Shared Backup Label Switched Path Restoration," Work in Progress.
[11] G. Li, C. Kalmanek, J. Yates, G. Bernstein, F. Liaw, and V. Sharma, "RSVP-TE Extensions For Shared-Mesh Restoration in Transport Networks", Work in Progress.
[12] P. Pan (Editor), D.H. Gan, G. Swallow, J. Vasseur, D. Cooper, A. Atlas, and M. Jork, "Fast Reroute Extensions to RSVP-TE for LSP Tunnels", Work in Progress.
[13] A. Atlas, C. Villamizar, and C. Litvanyi, "MPLS RSVP-TE Interoperability for Local Protection/Fast Reroute", Work in Progress.
[14] A. Chiu and J. Strand, "Joint IP/Optical Layer Restoration after a Router Failure", Proc. OFC'2001, Anaheim, CA, March 2001.
[15] K. Kompella and Y. Rekhter, "Multi-area MPLS Traffic Engineering", Work in Progress.
[16] G. Ash, et. al., "Requirements for Multi-Area TE", Work in Progress.
[17] A. Iwata, N. Fujita, G.R. Ash, and A. Farrel, "Crankback Routing Extensions for MPLS Signaling", Work in Progress.
[18] C-Y Lee, A. Celer, N. Gammage, S. Ghanti, G. Ash, "Distributed Route Exchangers", Work in Progress.
[19] C-Y Lee and S. Ghanti, "Path Request and Path Reply Message", Work in Progress.
[20] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M. and J. McManus, "Requirements for Traffic Engineering Over MPLS", RFC 2702, September 1999.
[21] Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998.

8. Acknowledgments 🫶

a lot of the direction in this document, and in the team's initial effort, was steered by the insightful questions from Bala Rajagoplan, Greg Bernstein, Yangguang Xu, and Avri Doria 🧠✨. that set of questions is attached as Appendix A in this document 📎

after the first draft was released, a bunch of comments came in 💌. thanks to the inputs from Jerry Ash, Sudheer Dharanikota, Chuck Kalmanek, Dan Koller, Lyndon Ong, Steve Plote, and Yong Xue 💖

9. Contributing Authors 🌟

Jim Boyle (PDNets), Rob Coltun (Movaz), Tim Griffin (AT&T), Ed Kern, Tom Reddington (Lucent) and Malin Carlzon 💜

Appendix A: Questions used to help develop requirements ✨

A. Definitions 💖

Okay so bestie, first things first: the design team has to nail down what "survivability", "restoration", "protection", "protection switching", "recovery", "re-routing" etc. actually mean, and how they relate to each other 🧠. That way the requirements doc can say super precisely which of these it's covering. In this appendix, "restoration" is used as the umbrella term for the whole broad set of policies and mechanisms that keep things surviving 🦋.

B. Network types and protection modes 🌈

B.1. Network types

Okay, so what's the scope when it comes to the types of networks? Specifically, are these in scope:
- Restoration of connections in mesh optical networks (opaque or transparent)
- Restoration of connections in hybrid mesh-ring networks
- Restoration of LSPs in MPLS networks (composed of LSRs overlaid on a transport network, e.g., optical)
- Any other types of networks?
- Is commonality of approach, or optimization of approach more important?

B.2. Protection modes

What are the requirements for the protection modes that should be supported in each network type covered? (Examples of protection modes include 1+1, M:N, shared mesh, UPSR, BLSR, newly defined modes such as P-cycles, etc.) 🎀

B.3. Local and end-to-end protection

What are the requirements on local span (i.e., link by link) protection and end-to-end protection, and the interaction between them? E.g.: what should be the granularity of connections for each type (single connection, bundle of connections, etc). 🫶

C. Hierarchy 🌸

C.1. Vertical (between two network layers)

What are the requirements for the interaction between restoration procedures across two network layers, when these features are offered in both layers? (Example, MPLS network realized over pt-to-pt optical connections.) Under such a case, 😭

(a) Are there any criteria to choose which layer should provide protection? 👑

(b) If both layers provide survivability features, what are the requirements to coordinate these mechanisms? 🔥

(d) Would the benefits be worth additional complexity associated with routing isolation (e.g. VPN, areas), security, address isolation and policy / authentication processes? 🙈

C.2. Horizontal (between two areas or administrative subdivisions within the same network layer)

Okay, so horizontal means between two areas or administrative subdivisions in the same network layer:

(a) What are the criteria that trigger the creation of protocol or administrative boundaries pertaining to restoration? (e.g., scalability? multi-vendor interoperability? what are the practical issues?) multi-provider? Should multi-vendor necessitate hierarchical separation? 🍓

When such boundaries are defined:

(b) What are the requirements on how protection/restoration is performed end-to-end across such boundaries? 🧚

(c) If different restoration mechanisms are implemented on two sides of a boundary, what are the requirements on their interaction? 💜

What is the primary driver of horizontal hierarchy? (select one) ⭐

functionality (e.g. metro -v- backbone)
routing scalability
signaling scalability
current network architecture, trying to layer on TE on top of an already hierarchical network architecture
routing and signalling

For signalling scalability, is it:

manageability
processing/state of network
edge-to-edge N^2 type issue

For routing scalability, is it:

processing/state of network
are you flat and want to go hierarchical
or already hierarchical?
data or TDM application?

D. Policy 🦋

What are the requirements for policy support during protection/restoration, e.g., restoration priority, preemption, etc. 👑

E. Signaling Mechanisms 💅

What are the requirements on the signaling transport mechanism (e.g., in-band over SDH/SONET overhead bytes, out-of-band over an IP network, etc.) used to communicate restoration protocol messages between network elements? What are the bandwidth and other requirements on the signaling channels? 🌈
What are the requirements on fault detection/localization mechanisms (which is the prelude to performing restoration procedures) in the case of opaque and transparent optical networks? What are the requirements in the case of MPLS restoration? 🔥
What are the requirements on signaling protocols to be used in restoration procedures (e.g., high priority processing, security, etc)? 🫶
Are there any requirements on the operation of restoration protocols? 🧚

F. Quantitative ⭐

What are the quantitative requirements (e.g., latency) for completing restoration under different protection modes (for both local and end-to-end protection)? 🍓

G. Management 💜

What information should be measured/maintained by the control plane at each network element pertaining to restoration events? 🌸
What are the requirements for the correlation between control plane and data plane failures from the restoration point of view? 🙈

Editors' Addresses 📮

Wai Sum Lai AT&T 200 Laurel Avenue Middletown, NJ 07748, USA

Phone: +1 732-420-3712 EMail: wlai@att.com

Dave McDysan WorldCom 22001 Loudoun County Pkwy Ashburn, VA 20147, USA

EMail: dave.mcdysan@wcom.com

Full Copyright Statement 🎀

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 💖

Funding for the RFC Editor function is currently provided by the Internet Society. ✨

📦 where this page comes from

ietf: rfc3386 2002-11
💖 girlified by claude-haiku-5-5 on 2026-10-16: a language model's rewrite of the original, under its terms