RFC 7654

Benchmarking Methodology for In-Service Software Upgrade (ISSU)

📅 published October 2015, from ietf

RFC 7654 🦋 Benchmarking Methodology for In-Service Software Upgrade (ISSU) 💅

okay bestie, this is the RFC that gives you a way to measure how well a device handles a software upgrade while it's still running 🫶 Not just "did it work?" but "how much did it wobble?" 📊✨

Quick title-block tea ☕ This is an Informational doc, so it is not a standards-track spec 🎀 It was published in October 2015 and it's ISSN 2070-1721 🌸 The authors are S. Banks (VSS Monitoring), F. Calabria (Cisco Systems), G. Czirjak and R. Machat (Juniper Networks) 👑 It doesn't obsolete or update any other RFC, so it's a fresh start, no drama 💜

The whole vibe is: ISSU is when a router or switch swaps out its software on the fly, and everyone wants that to be invisible 🦋 Spoiler: nobody gets zero downtime, but the goal is "as close to zero as possible" 🍓

Abstract

Modern forwarding devices try super hard to keep control-plane and data-plane disruptions to a minimum during planned software changes, using a trick called In-Service Software Upgrade (ISSU) 🔥 This doc gives everyone a shared set of methods and procedures to characterize how a Device Under Test (DUT) behaves during an ISSU event 💖

Status of This Memo

Okay so this is not a standards track spec, it's just published for info 🧚 It is an IETF product though, so it reflects consensus from the IETF community and got public review and IESG approval 🫶 Not every IESG-approved doc is on the path to becoming an Internet Standard tho, so check Section 2 of RFC 5741 if you're curious 🤓

Want the latest status, errata, or a way to leave feedback? Head to http://www.rfc-editor.org/info/rfc7654 💌

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as of the publication date. Read those carefully, they spell out your rights and restrictions 📜 Any Code Components pulled from this doc need Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, per that same license 💅

Table of Contents

1. Introduction ....................................................3
2. Conventions Used in This Document ...............................4
3. Generic ISSU Process, Phased Approach ...........................4
   3.1. Software Download ..........................................5
   3.2. Software Staging ...........................................6
   3.3. Upgrade Run ................................................6
   3.4. Upgrade Acceptance .........................................7
4. Test Methodology ................................................7
   4.1. Test Topology ..............................................7
   4.2. Load Model .................................................8
5. ISSU Test Methodology ...........................................9
   5.1. Pre-ISSU Recommended Verifications .........................9
   5.2. Software Staging ...........................................9
   5.3. Upgrade Run ...............................................10
   5.4. Post-ISSU Verification ....................................11
   5.5. ISSU under Negative Stimuli ...............................12
6. ISSU Abort and Rollback ........................................12
7. Final Report: Data Presentation and Analysis ...................13
   7.1. Data Collection Considerations ............................14
8. Security Considerations ........................................15
9. References .....................................................15
   9.1. Normative References ......................................15
   9.2. Informative References ....................................16
Acknowledgments ...................................................16
Authors' Addresses ................................................16

1. Introduction ✨

Okay, so most Service Provider (SP) network operators need ISSU 💅 Modern forwarding devices support it so they can upgrade or downgrade from one software version to another, with the goal of killing router downtime and service outages 🔥 Real talk though, most operators really want zero downtime, but "minimal downtime and service degradation" is usually what they actually expect 🫶

The ISSU operation can be an atomic version change of the whole system software, or something more modular, like a patch or maintenance upgrade 🧚 This procedure can check either approach, depending on what the vendor hardware and software support 🌸

Here's the behavior people want from an ISSU operation:

  • The software gets migrated from one version to a successive version, or the other way around 🎀
  • There are no control-plane interruptions through the whole process. The upgrade or downgrade happens while the device stays "in service" 💖 Still, most service providers will do this in a maintenance window, even with redundancy, just to be safe 🛡️
  • Forwarding-plane interruptions are minimal to none 🦋
  • The total upgrade time is kept as short as possible, since a long window means more exposure to outages (like an external failure hitting a network that's running with less redundancy) 😭

This doc gives you procedures to measure a forwarding device's ISSU behavior with numbers, based on those expectations 📊✨

Different hardware setups can be benchmarked, but a typical forwarding device that supports ISSU has at least one pair of Routing Processors (RPs) running redundantly, plus single or multiple forwarding engines (line cards) that may or may not be redundant, along with fabric cards or other parts as needed 💜 A device might also do ISSU by upgrading independent process components without affecting the rest of the box 🧚 For example, the software module for SNMP could be upgraded without touching other operations 🌈

The multi-chassis deployment concept can be characterized with these methodologies too, but the implementation details (like process placement and such) are out of scope for this doc 🫶

Since most modern forwarding devices that need ISSU have redundant RPs and separate control-plane and data-plane hardware, this doc focuses on methods that apply directly to those platforms 🔥 The ideas here should be easy to extend to other device architectures too 🦋

2. Conventions Used in This Document ✨

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 📜

These words only carry that RFC 2119 meaning when they're in ALL CAPS 💅 Lowercase uses don't count 🙈

3. Generic ISSU Process, Phased Approach ✨

ISSU is basically how a device behaves when someone plans a change to its software 🦋 That could mean the core OS, separate processes or daemons, or even firmware in programmable hardware like a Complex Programmable Logic Device (CPLD) or Field-Programmable Gate Array (FPGA) 💖 The whole goal is to make those changes with minimal or no disruption to the device's main job 🫶

ISSU can be kicked off by a person typing at the device, or triggered automatically from a management system or the device itself 🧚 This doc focuses on the version where a user starts the ISSU by hand 💅

ISSU can be split into phases or activities, defined below 🌸 For each phase, the test operator must record the outcome and any relevant observations (more on those later in the doc) 📝 Heads up: a given vendor implementation might or might not let you abort an in-progress ISSU at certain stages 😭 There can also be restrictions on ISSU depending on the configuration. For example, ISSU with Bidirectional Failure Detection (BFD) [RFC5880] may not be supported 🙈 It's on the test operator to make sure the DUT is set up properly for the test environment 🛡️ Like any well-run test effort, the test plan should reflect these details and be written with the expected production environment in mind 💜 Analyzing the results of each phase together characterizes the whole ISSU process, with the main goal of spotting and quantifying any service disruption (from both data- and control-plane views), so operators can plan maintenance with more precision 📊✨

3.1. Software Download

Okay so in this first phase, the requested software package may be downloaded to the router and is typically stored on the device 🎀 The download can happen automatically as part of the upgrade, or be started separately 🫶 Splitting it out lets an admin download new code inside or outside a maintenance window, and downloading and saving code to disk on the router is not expected to impact operations 🦋 If the software can be downloaded outside the actual upgrade, the administrator should do that, because download timing can skew results in ways that aren't comparable 📊 Internal compatibility checks may be run by the software on the DUT to verify the checksum of the downloaded files, plus any other pertinent checks 💖 Depending on the vendor, these may include 1) verifying that the downloaded modules meet identified prerequisites, like hardware or firmware compatibility or minimum software requirements, or even 2) making sure the device is "authorized" to run the target software 🔥

Where the product offers these mechanisms, the tester should verify them, with the goal of avoiding operational issues in production 🧚 Verification should cover both positive checks (making sure an ISSU action should be permitted) and negative tests (setting up scenarios where the checks would report exceptions) 💜

🫶 That's the first chunk, bestie! More parts are coming, so stay tuned 💅✨

3.2. Software Staging 📦✨

okay so phase two is where the requested software package actually gets loaded into the right parts of the forwarding device, usually the RP in standby state 🫶. The DUT can run its own internal compatibility checks as part of the upgrade itself, like verifying the checksum of the downloaded files and any other pertinent checks 🔍. Depending on the vendor, those checks might confirm that the downloaded modules meet a set of identified prerequisites, like hardware or firmware compatibility or minimum software requirements 💅. If the product offers these mechanisms, the tester should verify them too, with the same goal of avoiding operational issues in production 🙈. In that case, running these checks counts as part of the upgrade time and should be included in the testing results 📝. Once the new software is downloaded to the pertinent components of the DUT, the upgrade begins and the DUT starts getting itself ready 🚀. Depending on the vendor implementation, redundant hardware pieces in the DUT are expected to be upgraded too, including the backup or secondary RP 👑.

3.3. Upgrade Run 🎢

this phase is giving "switchover" energy 🦋. A switchover of RPs may take place, so that one RP is now running the new version of software. More importantly, the "Upgrade Run" phase is where the internal changes to information and state (stored on the router, on disk, and in memory) either get migrated to the "new" version of code, or get transformed or rebuilt to meet the standards of the new version of code, and then get pushed onto the right pieces of hardware 💜. Any outage(s) on the control or forwarding plane may be observed during this phase. This is the critical phase of the ISSU, where the control plane should not be impacted and any interruptions to the forwarding plane should be minimal to none 😭.

If any control- or data-plane interruptions show up during this stage, they should be recorded as part of the results document 📋.

For some implementations, the two stages described in Section 3.2 and above may be concatenated into one monolithic operation. In that case, the calculation of the respective ISSU time intervals may need to be adapted accordingly 🧮.

3.4. Upgrade Acceptance ✅

in this phase, the new version of software must be running on all the physical nodes of the logical forwarding device (RPs and line cards as applicable) 🔥. At this point, configuration control goes back to the operator, and normal device operation (i.e., outside of ISSU-oriented operation) resumes 🌸.

4. Test Methodology 🧪💖

As stated by [RFC6815], the Test Topology Setup must be part of an Isolated Test Environment (ITE) 🫧.

The reporting of results must take into account the repeatability considerations from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple trials and report average results. The results get reported in a simple statement that includes the measured frame loss and ISSU impact times 📊.

4.1. Test Topology 🗺️

The hardware configuration of the DUT should be identical to the one expected to be, or currently deployed, in production so the benchmark actually means something 💅. This includes the number of RPs, hardware version, memory, and initial software release, any common chassis components (such as fabric hardware in the case of a fabric-switching platform), and the specific line cards (version, memory, interface type, rate, etc.) 🧩.

For the control and data plane, different configuration approaches may be used. The recommended approach is "mimicking" the existing production data- and control-plane information, so you can emulate all the necessary Layer 1 through Layer 3 communications and, where appropriate, the upper-layer characteristics of the network, plus the end-to-end traffic/communication pairs 🪞. In other words, design a representative load model of the production environment and deploy a collapsed topology using test tools and/or external devices, where the DUT will be tested. Heads up: the negative impact of ISSU operations is likely to hit scaled, dynamic topologies harder than simpler, static environments 🙈. So this production-based methodology is advised for most test scenarios ✨.

The second, more simplistic approach is to deploy an ITE where endpoints are "directly" connected to the DUT. Control-plane information stays at a minimum (only connected interfaces), and only a basic data plane of sources and destinations is applied 🌱. If you go this route, be careful: the systemic behavior of the ITE may not match what a device sees in a production network role. Control-plane validation may be minimal to none with this method 😬. So if you choose this approach, comparing against at least one production configuration is recommended, so you understand how directly relevant (and limited) the test exercise is 🎀.

4.2. Load Model 📈

In consideration of the defined test topology, a load model must be developed to exercise the DUT while the ISSU event is introduced 🏋️. The applied load should be defined so it gives a granular, repeatable verification of the ISSU impact on transit traffic 🔬. Enough traffic load (rate) should be applied to permit timing extrapolations at a minimum granularity of 100 milliseconds, e.g., 100 Mbps for a 10 Gbps interface ⏱️. Steady traffic streams are preferred over bursty loads, since they're way easier to analyze 🧁.

The traffic should be patterned to cover a broad range of source and destination pairs that resolve to a variety of FIB (Forwarding Information Base) prefix lengths 🌈. If the production network includes multicast traffic or VPNs (L2, L3, or IPsec), it is critical to include these in the model 👑.

For mixed protocol environments (e.g., IPv4 and IPv6), frames should be distributed between the different protocols, and that distribution should approximate the network conditions of deployment 🧃. In all cases, the details of the mixed protocol distribution must be included in the reporting 📝.

The feature, protocol timing, and other relevant configurations should be matched to the expected production environment. Sometimes the test operator may need to deviate from the production templates (for example, certain features may not support ISSU, or the test bed may not be able to accommodate them) 🙃. But the impact of any divergence should be clearly understood, and the differences must be recorded in the results documentation 🗒️. It is recommended that a Network Management System (NMS) be deployed, preferably similar to the one used in production. That lets you monitor the DUT while it is being tested, both to support the impact analysis on system resources and to detect interference with non-transit (management) traffic caused by the ISSU operation 👀. It is suggested that the actual test exercise be managed through direct console access to the DUT, if at all possible, so a network interruption can't wreck the test run 🫶.

All in all, the load model should simulate the production network environment as closely as possible so the results actually apply 💖.

5. ISSU Test Methodology 🧫

As previously described, for the purposes of this test document, the ISSU process is divided into three main phases 🍰. The following methodology assumes a suitable test topology has been built per Section 4. The methodology for each of the phases above follows right after 💅.

The steps of this phase go like this:

  1. Verify that enough hardware and software resources are available to complete the Load operation (e.g., enough disk space) 💾.
  2. Verify that the redundancy states between RPs and other nodes are as expected (e.g., redundancy on, RPs synchronized) 🤝.
  3. Verify that the device, if it's running protocols capable of NSR (Non-Stop Routing), is in a "ready" state. That means the sync between RPs is complete and the system is ready for failover, if necessary 🚦.
  4. Gather a configuration snapshot of the device and all of its applicable components 📸.
  5. Verify that the node is operating in a "steady" state (that is, no critical or maintenance function is currently being performed) 🧘.
  6. Note any other operational characteristics the tester may deem applicable to the specific implementation deployed 🗒️.

5.2. Software Staging 🧪

The steps of this phase go like this:

  1. Establish all relevant protocol adjacencies and stabilize routing within the test topology. In particular, make sure the scaled levels of the dynamic protocols are dimensioned as specified by the test topology plan 🏗️.
  2. Clear relevant logs and interface counters to simplify analysis. If possible, set logging timestamps to a highly granular mode. If the topology includes management systems, make sure the appropriate polling levels are applied, sessions are established, and the responses are as expected 🧹.
  3. Apply the traffic loads specified in the load model previously developed for this exercise 🚛.
  4. Document an operational baseline for the test bed, with relevant data supporting the steps above (include all relevant load characteristics of interest in the topology, e.g., routing load, traffic volumes, memory and CPU utilization) 📉.
  5. Note the start time (T0) and begin the code change process using the mechanisms expected to be used in production (e.g., active download with TFTP, FTP, SCP, etc., or direct install from local or external storage facility). To make sure the ISSU process timings aren't skewed by the lack of a network-wide synchronization source, using a network NTP source is encouraged ⏰.
  6. Take note of any logging information and command-line interface (CLI) prompts as needed. (This detail will be vendor specific.) Respond to any DUT prompts in a timely manner 💬.
  7. Monitor the DUT for the reload of the secondary RP to the new software level. Once the secondary has stabilized on the new code, note the completion time. The duration of these steps will be recorded as "T1" 🏁.
  8. Review system logs for anomalies, check that relevant dynamic protocols have remained stable, and note any traffic loss. Verify that deployed management systems haven't identified any unexpected behavior 🕵️‍♀️✨.

5.3. Upgrade Run 🚀

okay so this bit assumes the software load step and the upgrade step can be controlled separately, bestie 💅 if they can't, keep that timer running and watch for the ISSU to finish like we talked about below ⏱️

  1. Note the start time, then kick off the actual upgrade procedure 🎬
  2. Watch the secondary route processor as it boots up on the new software and takes over mastership of the DUT 👀 this is the moment to really look for control-plane disruption, traffic impact, or any other weird behavior 😬 once the DUT has converged on the new code and is back to normal, note the completion time and log how long this step took as "T2" ✨
  3. Dig through the syslog data on the DUT and its neighbors for anything that would be disruptive in production 🕵️‍♀️ (line card reloads, control-plane flaps, etc.) Then check the traffic generators for any traffic loss during this window. If the Test Set reported traffic loss, log the number of frames lost as "TPL_frames", where TPL stands for "Total Packet Loss" 📉 If the Test Set also gives outage duration, log that as "TPL_time". (Or, if you'd rather, TPL_time can be calculated as (TPL / Offered Load) * 1000. Offered Load is in packets per second, and TPL_time comes out in milliseconds 🧮)
  4. Check the DUT status against whatever NMS is managing the DUT and its neighbors 🖥️ Write down the CPU and memory stats both during and after the ISSU upgrade event, and make sure memory and CPU have settled back to the level you baselined earlier 🧘‍♀️

5.4. Post-ISSU Verification 🔍

Okay, this is a set of post-ISSU checks that aren't technically part of the ISSU process itself, but they're recommended so you can prove the upgrade actually went well 💖

  1. Configuration delta analysis

Look at the post-ISSU configs and figure out whether anything changed, either because someone messed up the process or because the upgraded code behaves differently 🧐

  1. Exhaustive control-plane analysis

Review the RIB and FIB details to see whether any unexpected changes crept into the forwarding paths 🌊

  1. Verify that both RPs are up, and that the control-plane redundancy mechanism is enabled and fully synchronized 🫶
  2. Verify that no control-plane (protocol) events or flaps were detected 🙌
  3. Verify that no L1 and or L2 interface flaps were observed 🔌
  4. Document whether the operation was hitless or whether there was an outage, based on the counter values the Test Set gave you 📝

5.5. ISSU under Negative Stimuli 🌪️

This one is OPTIONAL 🎀 but it's giving main character energy: the operator can run an ISSU test while the DUT is under stress from route churn during any or all of the phases of the ISSU process 😤

There are a couple of ways to set this up. One approach has the operator pull statistical info from the production environment and then pick a specific number of routes to flap every "fixed" or "variable" interval 📊 Or the operator can just preselect a fixed number of prefixes to flap. For example, an operator might flap 1% of all the BGP routes every minute, then restore them 1 minute later 🔁 The tester might apply this stress across the entire ISSU process or, most importantly, during the run phase 🏃‍♀️ Super important: these stress routes must not overlap with the ones the load model uses to calculate TPL_time (the recorded outage) 🙅‍♀️ Also, there should not be any "operator-induced" control-plane protocol adjacency flaps for the whole test, because they can mess up the results of the entire exercise 💀 For example, triggering IGP adjacency events may force routing tables to be recomputed, which affects the ISSU timings people see 🫠 It's not recommended, but if the test operator really wants those trigger events, be careful not to let unexpected anomalies sneak into the test harness 🧪

6. ISSU Abort and Rollback 🙈

If a vendor supports it, the operator can abort the ISSU process for any reason 🛑 The end result and behavior can depend on the phase where the abort happened, though. This is implementation dependent, but as a general recommendation: if you abort during "Software Download" or "Software Staging", you shouldn't see any impact to service or device functionality ✅ But if you abort during "Upgrade Run" or "Upgrade Accept", the system may reload and go back to the previous software release, so that can be service affecting 😬 If vendor support exists, verify the abort/rollback functionality, then quantify any impact, following the procedures above 🧾

7. Final Report: Data Presentation and Analysis 📊

Every ISSU impact result gets wrapped up in one simple statement about the "ISSU Disruption Impact", including the measured frame loss and impact time. Impact time is the time frame from the TPL_time reported outage ✨ These are the main data points everyone cares about 👑

But the whole ISSU operational impact matters too, especially when planning maintenance 🗓️ so there are some extra reporting points:

     Software download / secondary update      T1

     Upgrade/Run                               T2

     ISSU Traffic Disruption (Frame Loss)      TPL_frames

     ISSU Traffic Impact Time (milliseconds)   TPL_time

     ISSU Housekeeping Interval                T3

     (Time for both RPs up on new code and fully synced - Redundancy
     restored)

     Total ISSU Maintenance Window             T4 (sum of T1+T2+T3)

The results report must provide the following info 📋

  • DUT hardware and software detail
  • Test Topology definition and diagram (especially as related to the ISSU operation)
  • Load Model description including protocol mixes and any divergence from the production environment
  • Time Results as per above
  • Anomalies Observed during ISSU
  • Anomalies Observed in post-ISSU analysis

It is RECOMMENDED that the following parameters be reported as outlined below 🌸

Parameter                Units or Examples
---------------------------------------------------------------

Traffic Load             Frames per second and bits per second

Disruption (average)     Frames

Impact Time (average)    Milliseconds

Number of trials         Integer count

Protocols                IPv4, IPv6, MPLS, etc.

Frame Size               Octets

Port Media               Ethernet, Gigabit Ethernet (GbE),
                         Packet over SONET (POS), etc.

Port Speed               10 Gbps, 1 Gbps, 100 Mbps, etc.

Interface Encaps         Ethernet, Ethernet VLAN, PPP,
                         High-Level Data Link Control (HDLC), etc.

Number of Prefixes       Integer count

flapped (ON Interval)    (Optional)  # of prefixes / Time (min.)

flapped (OFF Interval)   (Optional)  # of prefixes / Time (min.)

Document any configuration deltas you see after the ISSU upgrade has taken effect 🔎 Call out differences caused by changes in the patch or release level, and also anything that's an aberrant change from a software fault 🐛 In either case, analyze any unexpected behavior change and decide what the impact is, whether that's functional variances or operational impacts on existing scripts or management mechanisms 🧑‍💻

7.1. Data Collection Considerations 🕵️

Okay, real talk: while a DUT is going through an ISSU, its own data collection and reporting (counters, interface stats, log messages, etc.) might not be accurate 🫣 So don't rely on the DUT's data collection methods. Instead, use the test tools and equipment to collect the data for the reporting in Section 7 💅 When you design or add new test cases, make sure the data you want can be collected from the test tools themselves, or from other external equipment outside the DUT 💖

8. Security Considerations 🔐

All BMWG memos are limited to testing in a laboratory Isolated Test Environment (ITE), which avoids accidentally interrupting production networks during testing 🧪 Nobody wants that drama 😭

All benchmarking is limited to technology characterization using controlled stimuli in a lab environment, with dedicated address space and the other constraints [RFC2544] 🎀

The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. This is the one rule I need you to really hear, bestie 🚨

Also, benchmarking is done on a "black-box" basis, relying only on measurements observable outside the Device Under Test / System Under Test (DUT/SUT) 🕶️

Special capabilities should not exist in the DUT/SUT just for benchmarking purposes 🙅‍♀️ Any security implications from the DUT/SUT should be identical in the lab and in production networks, no exceptions 🔥

9. References 📚

9.1. Normative References

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <http://www.rfc-editor.org/info/rfc2119>.

[RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
           Network Interconnect Devices", RFC 2544,
           DOI 10.17487/RFC2544, March 1999,
           <http://www.rfc-editor.org/info/rfc2544>.

9.2. Informative References

[RFC5880]  Katz, D. and D. Ward, "Bidirectional Forwarding Detection
           (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010,
           <http://www.rfc-editor.org/info/rfc5880>.

[RFC6815]  Bradner, S., Dubray, K., McQuaid, J., and A. Morton,
           "Applicability Statement for RFC 2544: Use on Production
           Networks Considered Harmful", RFC 6815,
           DOI 10.17487/RFC6815, November 2012,
           <http://www.rfc-editor.org/info/rfc6815>.

Acknowledgments 🫶

The authors wish to thank Vibin Thomas for his valued review and feedback 💜

Authors' Addresses 💌

Sarah Banks
VSS Monitoring
Email: sbanks@encrypted.net

Fernando Calabria
Cisco Systems
Email: fcalabri@cisco.com

Gery Czirjak
Juniper Networks
Email: gczirjak@juniper.net

Ramdas Machat
Juniper Networks
Email: rmachat@juniper.net
📦 where this page comes from