RFC 6201

Device Reset Characterization 🎀

okay bestie, this is RFC 6201 💅 It's an Informational doc from March 2011 (ISSN 2070-1721), so it's published for info and isn't a standards-track spec 🌸 It was written by R. Asati, C. Pignataro, F. Calabria (Cisco), and C. Olvera (Consulintel) ✨ It updates RFC 1242 and RFC 2544 and doesn't obsolete anything 🦋

---

Abstract ✨

okay so routers and switches sometimes need a restart 🔄, whether it's automatic or someone hits the button. we call that a "reset" in this doc 💖 and while it's happening, traffic can get interrupted, so it's really useful to know how long the device takes to start forwarding again ⏱️

this doc gives a methodology for characterizing resets (and reset time) when you're benchmarking forwarding devices 🧪 It's basically a clearer, more consistent set of reset test steps than what RFC 2544 gives you 💅 That's why it updates RFC 2544 🎀 It also defines the benchmarking term "reset time" 🦋 and it updates RFC 1242, but only for that term 🫶

---

Status of This Memo 🌸

This doc is not an Internet Standards Track spec; it's published for info purposes 💜

It's a product of the IETF 🌈 and it reflects IETF community consensus. It got public review and was approved by the IESG ✨ Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 if you're curious 🧐

For the current status, errata, or how to send feedback, look at http://www.rfc-editor.org/info/rfc6201 🔗

---

Copyright Notice 🌈

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved 💖

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info), as they were on the publication date 📜 Please actually read them, bestie, because they explain your rights and restrictions for this doc 🙈 Any Code Components pulled out of this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Simplified BSD License describes 🔥

---

Table of Contents 🧚

   1. Introduction ....................................................3
      1.1. Scope ......................................................3
      1.2. Reset Time .................................................4
      1.3. Reset Time Measurement Methods .............................5
      1.4. Reporting Format ...........................................6
   2. Key Words to Reflect Requirements ...............................7
   3. Test Requirements ...............................................7
   4. Reset Tests .....................................................8
      4.1. Hardware Reset Tests .......................................9
           4.1.1. Routing Processor (RP) / Routing Engine Reset .......9
           4.1.2. Line Card (LC) Removal and Insertion (REQUIRED) ....11
      4.2. Software Reset Tests ......................................12
           4.2.1. Operating System (OS) Reset (REQUIRED) .............12
           4.2.2. Process Reset (OPTIONAL) ...........................13
      4.3. Power Interruption Test ...................................14
           4.3.1. Power Interruption (REQUIRED) ......................14
   5. Security Considerations ........................................15
   6. Acknowledgments ................................................16
   7. References .....................................................16
      7.1. Normative References ......................................16
      7.2. Informative References ....................................16

---

1. Introduction ✨

okay so an operational forwarding device (or any of its parts) might need a restart for all sorts of reasons 🔄 That's the "reset" event again 💖 Since forwarding can get interrupted during a reset, we want to know how long the device takes to get forwarding back 🏃‍♀️ Basically, we're asking about the recovery time after the reset (see Section 1.2, "Reset Time") 🎯

but ngl, the answer isn't simple anymore 😭 Modern forwarding devices have a bunch of hardware upgrades (distributed forwarding, etc.) and software upgrades (graceful restart, etc.), and all of those change the recovery time after a reset 🧠

1.1. Scope 🌸

okay so this doc gives a methodology for characterizing resets (and reset time) when you're benchmarking forwarding devices 🧪 It makes reset procedures clearer and more consistent than what [RFC2544] says 💅 Software upgrades bring their own benchmarking headaches, so they're out of scope here 🙅‍♀️

other benchmarking docs can use these procedures too, like [RFC2544], [RFC5180], [RFC5695], etc. ✨ We also expect other protocol-specific benchmarking docs to point back here for reset recovery time 💖 Specific RIB and FIB scaling stuff is out of scope too (Routing Information Base and Forwarding Information Base, for the record) 📚 and it can get really complex to characterize 🫠 But other docs can look at specific dynamic protocols' scaling and how they interact, and they can build on the reset tests here 🦋

this doc updates Section 26.6 of [RFC2544] and defines "reset time," which updates [RFC1242] 🎀

it only focuses on the reset criterion of benchmarking, and it assumes this is helpful for [RFC5180], [RFC5695], and other IETF Benchmarking Methodology Working Group (BMWG) efforts 🌈

1.2. Reset Time 🧚

Definition

okay so reset time is the total time a device is considered out of operation 💀 It includes both the time to do the reset and the time to recover from it ⏱️

Discussion

after a reset or power-up, network devices might not accept and forward frames for a while 😭 That stretch of unavailability is super useful when you're evaluating devices 📊 Also, some network devices need a reset whenever certain setup variables change 🔧 If the reset period is really long, network managers might not want to change those variables on production networks 😬

the events in this doc are entire reset events, so the recovery period includes both the reset itself and the recovery afterward 💅 Some resets are atomic (like pressing a reset button) 🔘 Others (like power cycling) can be multiple actions with a recognized gap between them ⚡ Either way, the clock starts at the beginning of the event and stops when forwarding is fully recovered after the reset events finish 🏁

Measurement Units

time in milliseconds, with enough resolution to tell apart different trials and different implementations ⏱️ See Section 1.4 🎀

Issues

there are three kinds of resets: hardware resets, software resets, and power interruptions 🔌 See Section 4 💖

See Also

this definition updates [RFC1242] 🦋

1.3. Reset Time Measurement Methods 🎯

reset time is the window where traffic forwarding is temporarily interrupted after a reset event 🚫 Technically, it's the time during which one or more frames get lost 💀 This is similar to the "Loss of Connectivity Period" from [IGPConv], Section 4 🧠

there are two accepted ways to measure reset time 👑

1. Frame-Loss Method - This method needs test tool capability to monitor the number of lost frames 📉 You have to know the offered stream rate (frames per second) too, and then reset time is calculated with this equation:

                                Frames_lost (packets)
          Reset_time = -------------------------------------
                         Offered_rate (packets per second)

2. Timestamp Method - This method needs test tool capability to timestamp each frame ⏰ The test tool stamps every frame it sends and checks the timestamp on every frame it receives 👀 During the test, it records the timestamp of the last frame received before the reset interruption (Timestamp A) and the timestamp of the first frame after the interruption ends (Timestamp B) 🏁 Reset time is the difference between Timestamp B and Timestamp A 💖

the tester/operator MAY use either method depending on what the test tool can do 🛠️ BUT the Frame-Loss method SHOULD be used if the test tool can (a) count lost frames per stream and (b) send test frames even when the physical link is down 📉 And the Timestamp method SHOULD be used if the test tool can (a) timestamp each frame, (b) monitor the timestamp on received frames, and (c) only send frames when the physical link status is UP ⏰ So the tool's capabilities basically decide which method you use 🧚 If the tool supports both, the tester/operator SHOULD use whichever one gives more accuracy 🎯

1.4. Reporting Format 💅

every reset result gets reported in a simple statement that includes the frame loss (if it was measured) and the reset times 📝

for each test case, it is RECOMMENDED that you report these parameters in these units 📊

    Parameter                Units or Examples
 ---------------------------------------------------------------

    Throughput               Frames per second and bits per
                             second

    Loss (average)           Frames

    Reset Time (average)     Milliseconds

    Number of trials         Integer count

    Protocol                 IPv4, IPv6, MPLS, etc.

    Frame Size               Octets

    Port Media               Ethernet, Gigabit Ethernet (GbE),
                             Packet over SONET (POS), etc.

    Port Speed               10 Gbps, 1 Gbps, 100 Mbps, etc.

    Interface Encap.         Ethernet, Ethernet VLAN,
                             PPP, High-Level Data Link Control
                             (HDLC), etc.

for mixed protocol environments, frames SHOULD be spread across all the different protocols 🌈 The distribution MAY approximate the network conditions you'd see in deployment ✨ In every case, the details of the mixed protocol distribution MUST be included in the report 📋

also, the DUT (Device Under Test) or SUT (System Under Test) and the test bed setup MUST be listed 🧾 That covers provisioning, port and line-card arrangement, configuration, and any deployed methodologies that might affect the overall reset time 🔍 (Check the additional factors in Section 3 for more 🎀)

reporting of results MUST take the repeatability considerations from Section 4 of [RFC2544] into account 🔁 It's RECOMMENDED to run multiple trials and report the average results 🫶

2. Key Words to Reflect Requirements 🎀

okay so bestie, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this doc get read the way BCP 14 and RFC 2119 [RFC2119] say they do 💅. RFC 2119 is the thing that explains these words so Standards-Track docs are super clear about what they mean 🦋. BUT, and this is important 🫶, this document is NOT a Standards-Track document, so these words are just here for flavor and clarity, not as a full-on spec vibe ✨.

3. Test Requirements 🧪

Okay, so tests SHOULD first be set up so the forwarding state comes back without depending on anything outside the box 🌸. That means static address resolution and routing/forwarding config, NOT dynamic protocols. Then, once that's working, tests MAY be run using the dynamic protocols the forwarding state depends on 🦋, like dynamic Interior Gateway Protocols (IGP), Address Resolution Protocol (ARP), PPP Control Protocols, and so on 💖. The considerations in this section apply, periodt 💅.

Now, for consistency and fairness when you're benchmarking a bunch of different DUTs, the Network tester/operator MUST (a) use identical control and data plane information during testing 🎀 and (b) document and report any factors that may influence the overall time after reset/convergence 📝. Lowkey this is the main rule of the whole section 👑.

Some of these factors include the following 🌈:

1. Type of reset - hardware (line-card crash, etc.) versus software (protocol reset, process crash, etc.) or even complete power failures ⚡
2. Manual versus automatic reset 🙈
3. Scheduled versus non-scheduled reset 📅
4. Local versus remote reset 🛰️
5. Scale - Number of line cards present versus in use 🔌
6. Scale - Number of physical and logical interfaces 🧩
7. Scale - Number of routing protocol instances 🌐
8. Scale - Number of routing table entries 🗂️
9. Scale - Number of route processors available 🧠
10. Performance - Redundancy strategy deployed for route processors and line cards 🛡️
11. Performance - Interface encapsulation as well as achievable throughput [RFC2544] 📦
12. Any other internal or external factor that may influence reset time after a hardware or software reset 🫧

The reset time is one of the key characterization results reported after each test run 📊. Like, the reset time during a reset test event may be zero ✨, but there may still be effects on traffic, such as transient delay variation or increased latency 😭. That is NOT covered in this doc though, it's deemed outside the scope of this document 🙈. In this case, only "no loss" is reported 💀.

4. Reset Tests 💖

This section is all about the tests for characterizing how long DUTs (Devices Under Test) or SUTs (Systems Under Test) take to recover from a reset 🧚. There are three types of resets considered in this document 👑:

1. Hardware resets
2. Software resets
3. Power interruption

Different types of resets can have different impacts on forwarding behavior 🌸. For example, a software reset (of a routing process) might not interrupt forwarding at all 🦋, whereas a hardware reset (of a line card) most likely will 😭.

Section 4.1 covers various hardware resets 🔧, Section 4.2 covers various software resets 💻, and Section 4.3 covers power interruption tests ⚡. These sections define and characterize these resets 📚.

Also, since device-specific implementations may vary for hardware and software type resets, it is desirable to classify each test case as "REQUIRED" or "OPTIONAL" 🎀.

4.1. Hardware Reset Tests 🔧

A hardware reset test is a test designed to characterize the time it takes a DUT to recover from a hardware reset ⏱️.

A hardware reset generally involves re-initializing one or more physical components in the DUT, but NOT the entire DUT 🧩.

A hardware reset is executed by the operator 👩‍💻, for example by physically removing a hardware component 🔌, by pressing a reset button for the component 🔘, or by triggering it from the command line interface (CLI) 💻.

Reset procedures that do NOT require physically removing and reinserting a hardware component are RECOMMENDED 🌟. These include using the CLI or a physical switch or button 🫶. If such procedures can't be performed (e.g., because the platform doesn't support them or the test case calls for something else), human operation time SHOULD be minimized across platforms and test cases as much as possible ⏳. Variation in human operator time SHOULD also be minimized across different vendors' products as much as practical 🍓, by having the same person do the operation and by practicing it first 🎀. Additionally, the time between removal and insertion SHOULD be recorded and reported 📝.

For routers that do NOT contain separate Routing Processor and Line Card modules, the hardware reset tests are NOT performed since they aren't relevant 🙅‍♀️. Instead, the power interruption tests MUST be performed (see Section 4.3) in these cases 💖.

4.1.1. Routing Processor (RP) / Routing Engine Reset 🧠

The Routing Processor (RP) is the DUT module that is mainly in charge of Control Plane functions 👑.

4.1.1.1. RP Reset for a Single-RP Device (REQUIRED) 💅

Objective

To characterize the time needed for a DUT to recover from a Route Processor hardware reset in a single RP environment 🎯.

Procedure

First, make sure the RP is in a permanent state it will return to after the reset 🛠️. You can do some or all of these operational tasks: save the current DUT configuration, specify boot parameters, make sure the right software files are available, or do other OS or hardware-related stuff 🧰.

Second, make sure the DUT can forward traffic for at least 15 seconds before any test activities start ⏰. The traffic should use the minimum frame size possible on the media used in the testing 📦, and the rate should be enough for the DUT to hit its maximum forwarding throughput 🚀. This gives a finer granularity in the reset time measurement 🔍.

Third, do the Route Processor (RP) hardware reset right here 💥. This could mean physically removing the RP to re-insert it later, or triggering a hardware reset another way (e.g., command line interface, physical switch, etc.) 🔘.

Finally, wrap it up by recording the frame loss or timestamps (as reported by the test tool) and calculating the reset time (as defined in Section 1.3) 📊✨.

Reporting Format

The reporting format is defined in Section 1.4 📄.

4.1.1.2. RP Switchover for a Multiple-RP Device (OPTIONAL) 🔀

Objective

To characterize the time needed for the "secondary" Route Processor (sometimes called the "backup" RP) of a DUT to become active after a "primary" (or "active") Route Processor hardware reset 🦋. This is often called "RP Switchover" 🫧. The characterization in this test SHOULD be done for the default DUT behavior and, if it exists, for the DUT's non-default configuration that minimizes frame loss 💜.

Procedure

This test characterizes RP Switchover 🔬. Lots of implementations have optimized switchover capabilities that minimize downtime during the actual switchover 🌸. This test has two sub-cases from a switchover standpoint 👯: first, a default behavior (with no switchover-specific configurations) and, potentially second, a non-default behavior with switchover configuration to minimize frame loss 💖. So the procedures here get run twice and reported separately 📑.

First, make sure the RPs are in a permanent state so the secondary RP will come up in the same state as the active RP 🧸. You can do some or all of these operational tasks: save the current DUT configuration, specify boot parameters, make sure the right software files are available, or do other OS or hardware-related stuff 🧰.

Second, make sure the DUT can forward traffic for at least 15 seconds before any test activities start ⏰. The traffic should use the minimum frame size possible on the media used in the testing 📦, and the rate should be enough for the DUT to hit its maximum forwarding throughput 🚀. This gives a finer granularity in the reset time measurement 🔍.

Third, do the primary Route Processor (RP) hardware reset right here 💥. This could mean physically removing the RP or triggering a hardware reset another way (e.g., command line interface, physical switch, etc.) 🔘. It's up to the operator whether the primary RP gets re-inserted after a grace period 🤷‍♀️.

Finally, wrap it up by recording the frame loss or timestamps (as reported by the test tool) and calculating the reset time (as defined in Section 1.3) 📊✨.

Reporting Format

The reset results MAY be reported twice 👯, once for the default switchover behavior (i.e., the DUT without any switchover-specific enhanced configuration) and once for the switchover-specific behavior if it exists (i.e., the DUT configured for optimized switchover capabilities that minimize downtime during the actual switchover) 🎀.

The reporting format is defined in Section 1.4 and also includes any specific redundancy scheme in place 🛡️.

4.1.2. Line Card (LC) Removal and Insertion (REQUIRED) 🔌

The Line Card (LC) is the DUT component that is responsible for packet forwarding 📦.

Objective

To characterize the time needed for a DUT to recover from a line-card removal and insertion event 🎯.

Procedure

For this test, the line card that is being hardware-reset MUST be on the forwarding path, and all destinations MUST be directly connected 🛤️.

First, complete some or all of these operational tasks: save the current DUT configuration, specify boot parameters, make sure the right software files are available, or do other OS or hardware-related stuff 🧰.

Second, make sure the DUT can forward traffic for at least 15 seconds before any test activities start ⏰. The traffic should use the minimum frame size possible on the media used in the testing 📦, and the rate should be enough for the DUT to hit its maximum forwarding throughput 🚀. This gives a finer granularity in the reset time measurement 🔍.

Third, do the Line Card (LC) hardware reset right here 💥. This could mean physically removing the LC to re-insert it later, or triggering a hardware reset another way (e.g., CLI, physical switch, etc.) 🔘.

Finally, wrap it up by recording the frame loss or timestamps (as reported by the test tool) and calculating the reset time (as defined in Section 1.3) 📊✨.

Reporting Format

The reporting format is defined in Section 1.4 📄.

4.2. Software Reset Tests 💻

A software reset test characterizes the time needed for a DUT to recover from a software reset ⏱️.

Unlike a hardware reset, a software reset involves only re-initializing the execution, data structures, and partial state within the software running on the DUT module(s) 🧠.

A software reset is initiated, for example, from the DUT's CLI 💻✨.

4.2.1. Operating System (OS) Reset (REQUIRED) ✨

Objective 🎯

Okay so the goal here is to figure out how long the DUT takes to bounce back after an operating system (OS) software reset. 💅

Procedure 📋

First, knock out some or all of these operational tasks: save the current DUT configuration, specify software boot parameters, make sure the right software files are available, or do any other operating system tasks. 🧁

Second, make sure the DUT can forward traffic for at least 15 seconds before any test activities start. The traffic should use the smallest frame size possible on the media used in testing, and the rate should be high enough for the DUT to hit its maximum forwarding throughput. This gives you a finer granularity in the reset time measurement. 🔍

Third, trigger an operating system re-initialization on the DUT with operational means, like the DUT's CLI or another management interface. 🚀

Finally, wrap it up by recording the frame loss or timestamps (as reported by the test tool) and calculating the reset time (as defined in Section 1.3). 🎀

Reporting Format 📝

The reporting format is defined in Section 1.4. ✨

4.2.2. Process Reset (OPTIONAL) 🧚

Objective 🎯

Here we're checking how long the DUT needs to recover from a software process reset. 🌸

That time period may depend upon the number and types of processes running in the DUT and which ones are tested. Different implementations of forwarding devices run different common processes, so it's giving variety. 🦋 A process reset should be performed only in the processes most relevant to the tester and most impactful to forwarding. Choose wisely, bestie! 💜

Procedure 📋

First, do some or all of these operational tasks: save the current DUT configuration, specify software parameters or environmental variables, or perform additional operating system tasks. 🫶

Second, make sure the DUT can forward traffic for at least 15 seconds before any test activities start. The traffic should use the minimum frame size possible on the media used in testing, and the rate should be sufficient for the DUT to attain the maximum forwarding throughput. This enables a finer granularity in the reset time measurement. ⭐

Third, trigger a process reset for each process running in the DUT that is considered for testing, from a management interface (e.g., by means of the CLI, etc.). 💅

Finally, finish the characterization by recording the frame loss or timestamps (as reported by the test tool) and calculating the reset time (as defined in Section 1.3). 👑

Reporting Format 📝

The reporting format is defined in Section 1.4 and is used for each process running in the DUT and tested. Since this test is so implementation-specific, the details of the actual process tested should be included along with the statement. 🧠

---

4.3. Power Interruption Test ⚡

"Power interruption" means a complete loss of power on the DUT. You can think of it as a special case of a hardware reset, triggered by losing power to the DUT or its components. It's characterized by re-initialization of all hardware and software in the DUT. 🔌

4.3.1. Power Interruption (REQUIRED) 💥

Objective 🎯

This one checks how long the DUT needs to recover from a complete loss of electric power, aka a complete power interruption. It simulates a total power failure or outage and should be indicative of the DUT/SUT's behavior during such an event. Major main character energy 😭🔥

Procedure 📋

First, make sure the entire DUT is in a permanent state it will return to after the power interruption. Do this by performing some or all of these operational tasks: save the current DUT configuration, specify boot parameters, make sure the appropriate software files are available, or perform additional operating system or hardware-related tasks. 🛠️

Second, make sure the DUT can forward traffic for at least 15 seconds before any test activities start. The traffic should use the minimum frame size possible on the media used in testing, and the rate should be sufficient for the DUT to attain the maximum forwarding throughput. This enables a finer granularity in the reset time measurement. 🦋

Third, interrupt the power (AC or DC) feeding the corresponding DUT's power supplies at this point. For example, you can physically remove the power supplies from the DUT to re-insert them later, or simply disconnect or switch off their power feeds (AC or DC, as applicable). The actual power interruption should last at least 15 seconds. ⏳

Finally, complete the characterization by recording the frame loss or timestamps (as reported by the test tool) and calculating the reset time (as defined in Section 1.3). 🎀

For easier comparison with other testing, 15 seconds are removed from the reported reset time. 🙈

Reporting Format 📝

The reporting format is defined in Section 1.4. 💖

---

5. Security Considerations 🔐

Benchmarking activities, as described in this document, are limited to technology characterization using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above. Lab-only, no cap 🧪

The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. Seriously, do NOT plug this into prod 😭🚨

Furthermore, benchmarking is performed on a "black-box" basis, relying solely on measurements observable externally to the DUT/SUT. 🖤

Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks. No secret benchmarking modes, bestie 🙅‍♀️💅

There are no specific security considerations within the scope of this document. ✨

6. Acknowledgments 🫶

The authors would like to thank Ron Bonica, who motivated them to write this document. The authors would also like to thank Al Morton, Andrew Yourtchenko, David Newman, John E. Dawson, Timmons C. Player, Jan Novak, Steve Maxwell, Ilya Varlashkin, and Sarah Banks for providing thorough review, useful suggestions, and valuable input. Iconic squad 👑🌈

7. References 📚

7.1. Normative References

[RFC1242] Bradner, S., "Benchmarking Terminology for Network
          Interconnection Devices", RFC 1242, July 1991.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
          Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
          Network Interconnect Devices", RFC 2544, March 1999.

7.2. Informative References

[IGPConv] Poretsky, S., Imhoff, B., and K. Michielsen, "Benchmarking
          Methodology for Link-State IGP Data Plane Route
          Convergence", Work in Progress, February 2011.

[RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D.
          Dugatkin, "IPv6 Benchmarking Methodology for Network
          Interconnect Devices", RFC 5180, May 2008.

[RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding
          Benchmarking Methodology for IP Flows", RFC 5695, November
          2009.

Authors' Addresses 💌

Rajiv Asati
Cisco Systems
7025-6 Kit Creek Road
Research Triangle Park, NC 27709
USA

EMail: rajiva@cisco.com


Carlos Pignataro
Cisco Systems
7200-12 Kit Creek Road
Research Triangle Park, NC 27709
USA

EMail: cpignata@cisco.com


Fernando Calabria
Cisco Systems
7200-12 Kit Creek Road
Research Triangle Park, NC 27709
USA

EMail: fcalabri@cisco.com


Cesar Olvera Morales
Consulintel
Joaquin Turina, 2
Pozuelo de Alarcon, Madrid, E-28224
Spain

EMail: cesar.olvera@consulintel.es

That's a wrap on RFC 6201, babe! 🌈💖✨