RFC 5673

Industrial Routing Requirements in Low-Power and Lossy Networks

📅 published October 2009, from ietf

Industrial Routing Requirements in Low-Power and Lossy Networks ✨

Okay bestie, first the header tea 🫖. This is RFC 5673, from the Network Working Group, dated October 2009. It's called Industrial Routing Requirements in Low-Power and Lossy Networks. The editors are K. Pister, Ed. from Dust Networks and P. Thubert, Ed. from Cisco Systems. The other authors are S. Dwars from Shell and T. Phinney, who is a consultant. The category is Informational, so it's giving "here's useful info," not "this is an Internet standard." The header doesn't list anything it obsoletes or updates, so there's no drama there 🙈

Abstract 💖

Cheap wireless devices are about to be everywhere, and that's going to level up the productivity and safety of industrial plants. Plant workers get more info about what's happening, so they can work more efficiently 🌸 This document is here to figure out what a routing protocol needs to do for industrial Low-power and Lossy Networks (LLNs) made of field devices.

Status of This Memo 📝

This memo is for the Internet community, and it's giving informational vibes. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited 🦋

Copyright (c) 2009 IETF Trust and the people listed as document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can find at http://trustee.ietf.org/license-info. That's the version that was in effect when this document was published. Please actually read these documents, because they explain your rights and restrictions for this document. Code components pulled out of this document have to include the Simplified BSD License text described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the BSD License 💅

Table of Contents 📚

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
     1.1.  Requirements Language  . . . . . . . . . . . . . . . . . .  3
   2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   3.  Overview . . . . . . . . . . . . . . . . . . . . . . . . . . .  4
     3.1.  Applications and Traffic Patterns  . . . . . . . . . . . .  5
     3.2.  Network Topology of Industrial Applications  . . . . . . .  9
       3.2.1.  The Physical Topology  . . . . . . . . . . . . . . . . 10
       3.2.2.  Logical Topologies . . . . . . . . . . . . . . . . . . 12
   4.  Requirements Related to Traffic Characteristics  . . . . . . . 13
     4.1.  Service Requirements . . . . . . . . . . . . . . . . . . . 14
     4.2.  Configurable Application Requirement . . . . . . . . . . . 15
     4.3.  Different Routes for Different Flows . . . . . . . . . . . 15
   5.  Reliability Requirements . . . . . . . . . . . . . . . . . . . 16
   6.  Device-Aware Routing Requirements  . . . . . . . . . . . . . . 18
   7.  Broadcast/Multicast Requirements . . . . . . . . . . . . . . . 19
   8.  Protocol Performance Requirements  . . . . . . . . . . . . . . 20
   9.  Mobility Requirements  . . . . . . . . . . . . . . . . . . . . 21
   10. Manageability Requirements . . . . . . . . . . . . . . . . . . 21
   11. Antagonistic Requirements  . . . . . . . . . . . . . . . . . . 22
   12. Security Considerations  . . . . . . . . . . . . . . . . . . . 23
   13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 25
   14. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25
     14.1. Normative References . . . . . . . . . . . . . . . . . . . 25
     14.2. Informative References . . . . . . . . . . . . . . . . . . 25

1. Introduction 🎀

Okay so IT (Information Technology) is already getting applied to industrial Control Technology (CT), and it's only going to keep happening. This works when the IT tech can be constrained enough by Service Level Agreements (SLA) or other small changes to meet what industrial CT needs. Then CT gets to borrow all the intellectual, experiential, and training investment that already went into those IT precursors 🧠 The reuse keeps happening because it saves so much intellectual, experiential, and training effort.

Following that logic, a lot of vendors are already extending or replacing their local fieldbus [IEC61158] technology with Ethernet and IP-based solutions. Some examples are Common Industrial Protocol (CIP) EtherNet/IP, Modbus/TCP, Fieldbus Foundation High Speed Ethernet (HSE), PROFInet, and Invensys/Foxboro FOXnet. At the same time, wireless, low-power field devices are showing up and letting users collect way more information and remotely manage way more control points ✨

IPv6 is basically the core tech where both trends meet. You can see that in the current [ISA100.11a] industrial Wireless Sensor Networking specification. It takes layers 1-4 that were built for other purposes, like the [IEEE802.15.4] PHY and MAC, IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs) [RFC4919], and UDP, and adapts them for industrial CT. The catch is that there aren't open standards for routing in Low-power and Lossy Networks (LLNs) yet, so even ISA100.11a leaves routing to proprietary methods 😭

So the goal of this document is to look at what the industrial environment needs from a routing protocol in IPv6-based Low-power and Lossy Networks (LLNs), so the next generation of Control Technology can be powered up 🔥

1.1. Requirements Language 💅

So the big caps words, "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL", mean what RFC 2119 [RFC2119] says they mean in this document 📖

2. Terminology 🧚

This document uses terms defined in the ROLL (Routing Over Low-power and Lossy networks) terminology document [ROLL-TERM]. It also refers to some industrial standards:

  • HART: Highway Addressable Remote Transducer. It's a group of specs for industrial process and control devices, run by the HART Communication Foundation (see [HART]). The latest version is HART7, which includes the additions for WirelessHART [IEC62591].
  • ISA: International Society of Automation, an ANSI-accredited standards-making society. ISA100 is an ISA committee whose charter includes defining a family of standards for industrial automation. [ISA100.11a] is a working group inside ISA100 that's working on a standard for monitoring and non-critical process control applications 🌈

3. Overview ⭐

Wireless, low-power field devices let industrial users collect a lot more information and remotely manage a lot more control points. Deploying these wireless devices will seriously improve the productivity and safety of plants, and it'll make plant workers more efficient too 🦋 IPv6 is seen as a key tech for the scalability and interoperability this space needs, and it keeps showing up in standards, products under development, and early deployments.

Cable is seen as the more proven, safer option, and the existing operational deployments have been stable for a long time. So wire isn't expected to be replaced by wireless anytime soon. The industry consensus is more like wireless will hugely expand what automation can do, by controlling devices that weren't connected before because of cost and/or deployment complexity. But for LLNs to get adopted in industrial settings, the wireless network needs three qualities: low power, high reliability, and easy installation and maintenance. The routing protocol used for LLNs matters a lot for hitting those goals 💜

Industrial automation splits into two distinct application spaces, called "process" or "process control" and "discrete manufacturing" or "factory automation". In industrial process control, the product is usually a fluid (oil, gas, chemicals, etc.). In factory automation or discrete manufacturing, the products are individual things (screws, cars, dolls). The two segments overlap a bit in products and systems, but they're surprisingly separate communities 🙈 Specs aimed at industrial process control tend to be more tolerant of network latency than factory automation needs.

Even with that 'process' vs 'discrete' difference, both plant types need the same thing: automating data collection that used to be done by hand, or never got collected at all. Examples are wireless sensors that report the state of a fuse, report the state of a luminary, report HVAC status, report vibration levels on pumps, report man-down, and so on 🔥

Other new application areas apply to both 'process' and 'discrete' plants. These involve mobile sensors that roam in and out of plants, like active sensor tags on containers or vehicles 🧳

Some, if not all, of these applications will need to run on the same low-power and lossy wireless network technology. That could mean several disconnected, autonomous LLNs connecting to multiple hosts while sharing the same ether. Interconnecting those networks might be wanted, if only to supervise channel and priority allocations, to fully synchronize, or to share path capacity within a set of physical network components. It might also be unwanted for practical reasons, such as cyber security concerns related to plant safety and integrity 🔐

Every application space wants battery-operated networks of hundreds of sensors and actuators talking to LLN access points. In an oil refinery, the total number of devices could go past one million. But the devices will be clustered into smaller networks that, in most cases, interconnect and report to an existing plant network infrastructure 🏭

Existing wired sensor networks in this space usually use communication protocols with low data rates. That ranges from 1200 baud (e.g., wired HART) up to the 100-200 kbps range for most of the others. The existing protocols are often master/slave with command/response 💖

3.1. Applications and Traffic Patterns 💅

okay so the industrial market sorts process applications into three broad categories and six classes 🎀 let's go bestie!

  • Safety 🚨
    • Class 0: Emergency action. Always a critical function 💀
  • Control 🎛️
    • Class 1: Closed-loop regulatory control. Often a critical function ✨
    • Class 2: Closed-loop supervisory control. Usually a non-critical function 🌸
    • Class 3: Open-loop control. The operator takes action and controls the actuator (human in the loop) 🧑‍💻
  • Monitoring 👀
    • Class 4: Alerting. Short-term operational effect (for example, event-based maintenance) 🔔
    • Class 5: Logging and downloading / uploading. No immediate operational consequence (e.g., history collection, sequence-of-events, preventive maintenance) 🦋

Safety-critical functions effect the basic safety integrity of the plant 🛡️ These normally dormant functions kick in only when process control systems, or their operators, have failed 😭 By design and by regular interval inspection, they have a well-understood probability of failure on demand in the range of typically once per 10-1000 years 🕰️

In-time deliveries of messages become more relevant as the class number decreases 🏃‍♀️💨

Note that for a control application, the jitter is just as important as latency, and it has a potential of destabilizing control algorithms 🙈 bestie this one is serious!!

Industrial users are interested in deploying wireless networks for the monitoring classes 4 and 5, and in the non-critical portions of classes 2 through 3 📡🌈

Classes 4 and 5 also include asset monitoring and tracking. These include equipment monitoring and are essentially separate from process monitoring 🔧 An example of equipment monitoring is the recording of motor vibrations to detect bearing wear 🛠️ However, similar sensors detecting excessive vibration levels could be used as safeguarding loops that immediately initiate a trip, and thus end up being class 0 😱

In the near future, most LLN systems in industrial automation environments will be for low-frequency data collection 📈 Packets containing samples will be generated continuously, and 90% of the market is covered by packet rates of between 1/second and 1/hour, with the average under 1/minute ⏱️ In industrial process, these sensors include temperature, pressure, fluid flow, tank level, and corrosion 🌡️💧 Some sensors are bursty, such as vibration monitors that may generate and transmit tens of kilobytes (hundreds to thousands of packets) of time-series data at reporting rates of minutes to days 📦✨

Almost all of these sensors will have built-in microprocessors that may detect alarm conditions 🧠 Time-critical alarm packets are expected to be granted a lower latency than periodic sensor data streams 🚨💖

Some devices will transmit a log file every day, again with typically tens of kilobytes of data 📝 For these applications, there is very little "downstream" traffic coming from the LLN access point and traveling to particular sensors 🙃 During diagnostics, however, a technician may be investigating a fault from a control room and expect to have "low" latency (human tolerable) in a command/response mode 🧑‍🔧💅

Low-rate control, often with a "human in the loop" (also referred to as "open loop"), is implemented via communication to a control room because that's where the human in the loop will be 🏢 The sensor data makes its way through the LLN access point to the centralized controller, where it is processed. The operator sees the information and takes action, and the control information is then sent out to the actuator node in the network 🎮

In the future, it is envisioned that some open-loop processes will be automated (closed loop), and packets will flow over local loops and not involve the LLN access point 🔮 These closed-loop controls for non-critical applications will be implemented on LLNs 🌟 Non-critical closed-loop applications have a latency requirement that can be as low as 100 milliseconds, but many control loops are tolerant of latencies above 1 second ⏳

More likely though is that loops will be closed in the field entirely 🌿 In such a case, having wireless links within the control loop does not usually present actual value 🙅‍♀️ Most control loops have sensors and actuators within such proximity that a wire between them remains the most sensible option from an economic point of view 💰 This "control in the field" architecture is already common practice with wired fieldbusses 🚌 An "upstream" wireless link would only be used to influence the in-field controller settings and to occasionally capture diagnostics 📶 Even though the link back to a control room might be wireless, this architecture reduces the tight latency and availability requirements for the wireless links 🫶

Closing loops in the field:

  • does not prevent the same loop from being closed through a remote multivariable controller during some modes of operation, while being closed directly in the field during other modes of operation (e.g., fallback, or when timing is more critical) 🦋
  • does not imply that the loop will be closed with a wired connection, or that the wired connection is more energy efficient even when it exists as an alternate to the wireless connection 🔋

A realistic future scenario is for a field device with a battery or ultra-capacitor power storage to have both wireless and unpowered wired communications capability (e.g., galvanically isolated RS-485) 🔌 The wireless communication is more flexible and, for local loop operation, more energy efficient 💜 The wired communication capability serves as a backup interconnect among the loop elements, but without a wired connection back to the operations center blockhouse 🏠 In other words, the loop elements are interconnected through wiring to a nearby junction box, but the 2 km home-run link from the junction box to the control center does not exist 🙌

When wireless communication conditions are good, devices use wireless for loop interconnect, and either one wireless device reports alarms and other status to the control center for all elements of the loop, or each element reports independently 📡✨ When wireless communications are sporadic, the loop interconnect uses the self-powered galvanically isolated RS-485 link, and one of the devices with good wireless communications to the control center serves as a router for those devices that are unable to contact the control center directly 🧚

The above approach is particularly attractive for large storage tanks in tank farms 🛢️ Devices may not all have good wireless visibility of the control center, and a home-run cable from the tank to the control center is undesirable due to the electro-potential differences between the tank location and the distant control center that arise during lightning storms ⚡ iykyk that's a whole vibe 😭

In fast control, tens of milliseconds of latency is typical 🏎️ In many of these systems, if a packet does not arrive within the specified interval, the system enters an emergency shutdown state, often with substantial financial repercussions 💸 For a one-second control loop in a system with a target of 30 years for the mean time between shutdowns, the latency requirement implies nine 9s of reliability (aka 99.9999999% reliability) 👑 Given such exposure, given the intrinsic vulnerability of wireless link availability, and given the emergence of control in the field architectures, most users tend not to aim for fast closed-loop control with wireless links within that fast loop 🫠💅

3.2. Network Topology of Industrial Applications ✨

okay so network topology is lowkey hard to generalize 💅 but the majority of existing apps can be handled by networks of 10 to 200 field devices with a maximum of 20 hops 🦋 It's assumed the field devices will provide their own routing, so extra repeaters/routers won't be needed in most cases 🫶

For the vast majority of industrial apps, the traffic is mostly real-time publish/subscribe sensor data (also referred to as buffered) going from the field devices over an LLN toward one or more sinks 📡 Over time these sinks will increasingly be part of a backbone, but right now they're often fragmented and isolated 😭

The wireless sensor network (WSN) is an LLN of field devices with two logical roles: field routers and non-routing devices 🌸 It's acceptable and even probable that the split of roles across the field devices changes over time to balance the cost of forwarding amongst the nodes ⚖️

To scale a control network in terms of density, one possible architecture is a backbone that works like a canopy and aggregates multiple smaller LLNs 🌈 The backbone is a high-speed infrastructure network that may interconnect multiple WSNs through backbone routers 🔥 Infrastructure devices can be connected to the backbone too. A gateway/manager that connects the backbone to the plant network of the corporate network can be seen as collapsing the backbone and the infrastructure devices into a single device that runs all the required logical roles 🧚 The backbone is likely to become an option in industrial networks, iykyk 👑

Typically these backbones connect to the 'legacy' wired plant infrastructure, known as the plant network or Process Control Domain (PCD) 🏭 These plant automation networks are segregated domain-wise from the office network or office domain (OD), which is itself typically segregated from the Internet 🙈

Sinks for LLN sensor data can live on the plant network (the PCD), the business network (the OD), and the Internet 💖 Apps close to existing plant automation, like wired process control and monitoring systems running on fieldbusses, that need high availability and low latencies, and that are managed by 'Control and Automation' departments, usually live on the PCD 🎀 Other apps like automated corrosion monitoring, cathodic protection voltage verification, or machine condition (vibration) monitoring, where one sample per week counts as over-sampling, would more likely deliver their sensor readings to the OD 📊 These apps are 'owned' by, e.g., maintenance departments 🔧

Then there are apps like third-party-maintained luminaries, or vendor-managed inventory systems, where a chemicals supplier needs access to tank level readings at his customer's site 🧪 Those are best served with direct Internet connectivity all the way to the sensor at his customer's site 🌐 Temporary 'babysitting sensors' deployed for just a few days, say during startup or troubleshooting, or for ad hoc measurement campaigns for research and development, are other examples where the Internet is where the wireless sensor data lands 🍓 The OD and PCD should preferably be circumvented in that case, if quick deployment without potentially impacting plant safety integrity is required 🚨

This multiple-domain, multiple-application connectivity creates a seriously big challenge 😤 Lots of different apps will all share the same medium, the ether, within the fence 🌬️ They'd preferably share the same frequency bands and the same protocols, and be synchronized to optimize coexistence challenges, yet still be logically segregated so no intolerable shortcuts get created between existing wired domains 🚧

Given this challenge, LLNs are best treated as sitting on yet another segregated domain, separate from all the other wired domains where conventional security is organized by perimeter 🛡️ Moving away from the traditional perimeter-security mindset means moving toward stronger end-device identity authentication 🔐 That way LLN access points can split the various wireless data streams and interconnect back to the appropriate domain (pending the gateways' establishment of the message originators' identity and trust) 💜

Similar considerations apply to whether multiple applications may or may not share routing devices and their potentially redundant bandwidth within the network 🦋 The challenge is to balance available capacity, required latencies, expected priorities, and (last but not least) available (battery) energy within the routing devices 🔋✨

3.2.1. The Physical Topology

There is no specific physical topology for an industrial process control network 🗺️

One extreme example is a multi-square-kilometer refinery where isolated tanks compose a farm spread across the surface of the plant 🏗️ Some tanks have power but most have no backbone connectivity 😬 A few hundred field devices are deployed to ensure global coverage using a wireless self-forming, self-healing mesh network that might be 5 to 10 hops across 🕸️ Local feedback loops and mobile workers tend to be only 1 or 2 hops away 🏃‍♀️ The backbone is in the refinery proper, many hops away, and even there, powered infrastructure is typically several hops away too 🥲 In that case, hopping to or from the powered infrastructure may often be more costly than the direct route 💸

In the opposite extreme case, the backbone spans all the nodes, and most nodes are in direct sight of one or more backbone routers 👀 Most communication between field devices and infrastructure devices, as well as field device to field device, happens across the backbone 🌟 From afar this model looks a lot like the WiFi ESS (Extended Service Set) 📶 But from a layer-3 (L3) perspective, the issues are the default (backbone) router selection and the routing inside the backbone, whereas the radio hop toward the field device is really just a simple local delivery 🎯

         ---------+----------------------------
                  |          Plant Network
                  |
               +-----+
               |     | Gateway             M : Mobile device
               |     |                     o : Field device
               +-----+
                  |
                  |      Backbone
            +--------------------+------------------+
            |                    |                  |
         +-----+             +-----+             +-----+
         |     | Backbone    |     | Backbone    |     | Backbone
         |     | router      |     | router      |     | router
         +-----+             +-----+             +-----+
            o    o   o    o     o   o  o   o   o   o  o   o o
        o o   o  o   o  o  o o   o  o  o   o   o   o  o  o  o o
       o  o o  o o    o   o   o  o  o  o    M    o  o  o o o
       o   o  M o  o  o     o  o    o  o  o    o  o   o  o   o
         o   o o       o        o  o         o        o o
                 o           o          o             o     o
                        LLN

             Figure 1: Backbone-Based Physical Topology

An intermediate case is shown in Figure 1, with a backbone that spans the Wireless Sensor Network in such a way that any WSN node is only a few wireless hops away from the nearest backbone router 🌈 WSN nodes are expected to organize into self-forming, self-healing, self-optimizing logical topologies that let them lean on the backbone when that's most efficient 💫

It must be noted that the routing function is expected to be so simple that any field device could take on the role of a router, depending on the self-discovery of the topology and the power status of its neighbors 🧠 On the other hand, only devices equipped with the right hardware and software combination could take on the role of an endpoint for a given purpose, such as a sensor or actuator 🎀

3.2.2. Logical Topologies

Most of the traffic over the LLN is publish/subscribe of sensor data going from the field device toward a sink, which can be a backbone router, a gateway, or a controller/manager 📥 The destination of the sensor data is an infrastructure device that sits on the backbone and is reachable via one or more backbone routers 🛰️

For security, reliability, availability, or serviceability reasons, it's often required that the logical topologies are not physically congruent over the radio network 🔐 That means they form logical partitions of the LLN 🧩 For instance, a routing topology set up for control should be isolated from a topology that reports the temperature and the status of the vents, if that second topology has lesser constraints for the security policy 🌡️ This isolation might be implemented as Virtual LANs and Virtual Routing Tables in shared nodes in the backbone, but it effectively corresponds to physical nodes in the wireless network 🧁

Since publishing data is the raison d'etre for most sensors, in some cases it makes sense to proactively build a set of routes between the sensors and one or more backbone routers, and to maintain those routes at all times ⏰ Also, because of the lossy nature of the network, the routing in place should attempt to propose multiple paths in the form of Directed Acyclic Graphs oriented toward the destination 🌊

In contrast with the general need to maintain default routes toward the sinks, the need for field device to field device (FD-to-FD) connectivity is very specific and rare 🤏 The traffic associated with it might still be of foremost importance though 🚨 FD-to-FD routes are often the most critical, optimized, and well-maintained routes 👑 A class 0 safeguarding loop requires guaranteed delivery and extremely tight response times ⚡ Both respecting the criteria in the route computation and the quality of the route maintenance are critical for the field devices' operation 🫡 Typically, a control loop uses a dedicated direct wire that has very different capabilities, cost, and constraints than the wireless medium, with a need to use a wireless path as a backup route only in case the wired path is lost 🔌

Since each FD-to-FD route computation has specific constraints in terms of latency and availability, it can be expected that the shortest path possible will often be selected, and that this path will be routed inside the LLN rather than via the backbone 🏃 Route lifetimes can range from minutes for a mobile worker to tens of years for a command and control closed loop ⏳ Finally, time-varying user requirements for latency and bandwidth will change the constraints on the routes, which might trigger a constrained route recomputation, a reprovisioning of the underlying L2 protocols, or both, in that order 🔄 For instance, a wireless worker may start a bulk transfer to configure or diagnose a field device 🛠️ A level sensor device may need to run a calibration and send a bulk file to a plant 📁✨

okay bestie, [ISA100.11a] picked IPv6 as its network layer 🎀 and honestly? iconic choice. The huge address space and the potential subnet size (up to 10K nodes in a plant deployment) is giving main character energy 👑. In the ISA100 model, industrial apps fall into four big service categories:

  1. Periodic data (aka buffered) 💖. This is data generated on a schedule, with a well understood bandwidth requirement that's both deterministic and predictable. Timely delivery is basically the whole job of a wireless sensor network, so permanent resources get assigned to keep that bandwidth available 🔥. Buffered data usually has a short time to live, and the newer reading obsoletes the previous one (the old number is literally trash 💀). Sometimes alarms are low-priority and just get repeated over and over. End-to-end latency matters less here than how regularly the data shows up for the plant application ⏰.
  2. Event data 🦋. This category covers alarms and aperiodic data reports with bursty bandwidth requirements. In certain cases, alarms are critical and need a priority service from the network 🚨.
  3. Client/Server 🧚. Lots of industrial apps are built on a client/server model and run a command response protocol. The bandwidth is often bursty. Round-trip latency for some legacy systems was based on how long it took to send tens of bytes over a 1200 baud link, so hundreds of milliseconds is typical (vintage, but iconic 🙈). These requests get statistically multiplexed over the LLN, and cost-based, fair-share, best-effort service is usually expected 🫶.
  4. Bulk transfer 🍓. This means blocks of data sent across multiple packets, with temporary resources assigned to hit a transaction time constraint. Transient resources are assigned for a limited time (tied to file size and data rate) so the bulk transfer service requirements get met 💅.

4.1. Service Requirements

okay, these are the service parameters that can affect routing decisions in a resource-constrained network 🧠:

  • Data bandwidth 🌊 - the bandwidth might be allocated permanently or for a period of time to a specific flow that usually has well-defined burstiness and throughput. Some bandwidth also gets statistically shared between flows in a best-effort fashion.
  • Latency ⏱️ - the time it takes for data to transit the network from source to destination. This can be expressed as a deadline for delivery. Most monitoring latencies are in seconds to minutes.
  • Transmission phase 🕰️ - process applications can be synced to wall clock time and need coordinated transmissions. A common coordination frequency is 4 Hz (250 ms).
  • Service contract type - revocation priority 📜 - LLNs have limited network resources that can vary with time, so the system can become fully subscribed or even over-subscribed. System policies decide how resources get allocated when things are over-subscribed. The choices are blocking and graceful degradation.
  • Transmission priority 🎯 - this is how limited resources inside field devices get split across multiple services. For transmissions, a device has to pick which packet in its queue goes out at the next transmission opportunity, and packet priority is one criterion for that pick. For reception, a device has to decide how to store a received packet. Field devices are memory-constrained and receive buffers can fill up, so packet priority decides which packets get stored or discarded.

The routing protocol MUST also support different metric types for each link, used to compute the path against some objective function (e.g., minimize latency), depending on the nature of the traffic 🛣️.

So yeah, that's why the ROLL routing infrastructure is REQUIRED to compute and update constrained routes on demand 🚀. Expect this model to get more popular for FD-to-FD connectivity, and for some FD-to-infrastructure-device connectivity, over time 📈.

Industrial application data flows between field devices are not necessarily symmetric 🙃. Asymmetrical cost and unidirectional routes are common for published data and alerts, which make up most of the sensor traffic. The routing protocol MUST be able to compute a set of unidirectional routes with potentially different costs, composed of one or more non-congruent paths.

As multiple paths get set up and a variety of flows head toward the same destination (for instance, a node acting as a sink for the LLN), an additional marking/tagging mechanism based on upper-layer information will be REQUIRED. This lets intermediate routers tell the flows apart and make the right routing decision using only the content of the IPv6 packet (e.g., use of DSCP, Flow Label) 🏷️✨.

4.2. Configurable Application Requirement

bestie, time-varying user requirements for latency and bandwidth might mean the underlying L2 protocols need reprovisioning 🛠️. A technician may start a query/response session or bulk transfer to diagnose or configure a field device. A level sensor device may need to run a calibration and send a bulk file to a plant 📂. The routing protocol MUST support the ability to recompute paths based on network-layer abstractions of underlying link attributes/metrics that may change dynamically 🔄.

4.3. Different Routes for Different Flows

Because different service categories have different service requirements, it's often desirable to have different routes for different data flows between the same two endpoints 🦋. For example, alarm or periodic data from A to Z may require path diversity with specific latency and reliability. A file transfer between A and Z may not need path diversity 📦. The routing algorithm MUST be able to generate different routes with different characteristics (e.g., optimized according to different costs, etc.) 💜.

Dynamic or configured states of links and nodes influence whether a given path can fulfill operational requirements like stability, battery cost, or latency 🔋. Constraints such as battery lifetime come from the application itself. Since industrial application data flows are typically well-defined and well-controlled, it's usually possible to estimate a router's battery consumption for a given topology 🧮.

The routing protocol MUST support the ability to (re)compute paths based on network-layer abstractions of upper-layer constraints, so the level of operation stays within required parameters 🎯. Such information MAY be advertised by the routing protocol as metrics, so routing algorithms can establish appropriate paths that fit the upper-layer constraints 🪄.

The network layer handles an IPv6 packet based on the standard properties and the settings of its header fields 🎀. These fields include the 3-tuple of Flow Label and the Source and Destination Address, which can identify a flow, plus the Traffic Class octet, which can influence the Per Hop Behavior in intermediate routers 💖.

An application MAY choose how to set those fields for each packet or for streams of packets, and the routing protocol specification SHOULD state how different field settings will be handled to make different routing decisions 🫶✨.

5. Reliability Requirements ✨

okay so LLN reliability is NOT just one thing 💅 it's a whole bundle of unrelated aspects:

  1. Availability of source-to-destination connectivity when the application needs it, expressed in number of successes divided by number of attempts.
  2. Availability of source-to-destination connectivity when the application might need it, expressed in number of potential failures / available bandwidth,
  3. Ability, expressed in number of successes divided by number of attempts to get data delivered from source to destination within a capped time,
  4. How well a network (serving many applications) achieves end-to-end delivery of packets within a bounded latency,
  5. Trustworthiness of data that is delivered to the sinks,
  6. and others depending on the specific case.

So quantifying reliability is basically plotting it on a three (or more) dimensional graph 🌈 Different applications have different requirements, and squishing reliability into one dimensional parameter, like 'reliability on my wireless network is 99.9%', often creates more confusion than clarity 🙈

The impact of not receiving sensor data because of sporadic network outages can be devastating if nobody notices 😭 However, if destinations that expect periodic sensor data or alarm status updates fail to get them, then these systems can automatically take appropriate actions that prevent dangerous situations 🫶 Depending on the wireless application, appropriate action ranges from initiating a shutdown within 100 ms, to using a last known good value for as much as N successive samples, to sending out an operator into the plant to collect monthly data the conventional way, i.e., some portable sensor, or paper and a clipboard 📋

The impact of receiving corrupted data, and not being able to detect that the received data is corrupt, is often even more dangerous 💀 Data corruption can come from random bit errors due to white noise, or from occasional bursty interference sources like thunderstorms or leaky microwave ovens, but also from conscious attacks by adversaries 😤

Another critical aspect for the routing is the capability to ensure maximum disruption time and route maintenance 🛣️ The maximum disruption time is the time it takes at most for a specific path to be restored when broken. Route maintenance ensures that a path that is monitored cannot stay disrupted for more than the maximum disruption time. Maintenance should also ensure that a path continues to provide the service for which it was established, for instance, in terms of bandwidth, jitter, and latency ✨

In industrial applications, availability is usually defined with respect to end-to-end delivery of packets within a bounded latency. Availability requirements vary over many orders of magnitude 📈 Some non-critical monitoring applications may tolerate an availability of less than 90% with hours of latency. Most industrial standards, such as HART7 [IEC62591], have set user availability expectations at 99.9%. Regulatory requirements are a driver for some industrial applications. Regulatory monitoring requires high data integrity because lost data is assumed to be out of compliance and subject to fines 🧾 This can drive up either availability or trustworthiness requirements.

Because LLN link stability is often low, path diversity is critical 🦋 Hop-by-hop link diversity is used to improve latency-bounded reliability by sending data over diverse paths.

Because data from field devices is aggregated and funneled at the LLN access point before it is routed to plant applications, LLN access point redundancy is an important factor in overall availability 🌸 A route that connects a field device to a plant application may have multiple paths that go through more than one LLN access point. The routing protocol MUST be able to compute paths of not-necessarily-equal cost toward a given destination so as to enable load-balancing across a variety of paths 💜 The availability of each path in a multipath route can change over time. Hence, it is important to measure the availability on a per-path basis and select a path (or paths) according to the availability requirements 🎯

6. Device-Aware Routing Requirements 🔋

Wireless LLN nodes in industrial environments are powered by a variety of sources ⚡ Battery-operated devices with lifetime requirements of at least five years are the most common. Battery-operated devices have a cap on their total energy, and typically can report an estimate of remaining energy, and typically do not have constraints on the short-term average power consumption. Energy-scavenging devices are more complex 🧠 These systems contain both a power-scavenging device (such as solar, vibration, or temperature difference) and an energy storage device, such as a rechargeable battery or a capacitor. So these systems have limits on both long-term average power consumption (which cannot exceed the average scavenged power over the same interval) as well as the short-term limits imposed by the energy storage requirements. For solar-powered systems, the energy storage system is generally designed to provide days of power in the absence of sunlight ☀️ Many industrial sensors run off of a 4-20 mA current loop, and can scavenge on the order of milliwatts from that source. Vibration monitoring systems are a natural choice for vibration scavenging, which typically only provides tens or hundreds of microwatts. Due to industrial temperature ranges and desired lifetimes, the choices of energy storage devices can be limited, and the resulting stored energy is often comparable to the energy cost of sending or receiving a packet rather than the energy of operating the node for several days 🥲 And of course, some nodes will be line-powered 🔌

Example 1: solar panel, lead-acid battery sized for two weeks of rain.

Example 2: vibration scavenger, 1 mF tantalum capacitor.

Field devices have limited resources 🥺 Low-power, low-cost devices have limited memory for storing route information. Typical field devices will have a finite number of routes they can support for their embedded sensor/actuator application and for forwarding other devices' packets in a mesh network slotted-link.

Users may strongly prefer that the same device have different lifetime requirements in different locations 📍 A sensor monitoring a non-critical parameter in an easily accessed location may have a lifetime requirement that is shorter and may tolerate more statistical variation than a mission-critical sensor in a hard-to-reach place that requires a plant shutdown in order to replace 👑

The routing algorithm MUST support node-constrained routing (e.g., taking into account the existing energy state as a node constraint). Node constraints include power and memory, as well as constraints placed on the device by the user, such as battery life 🔋✨

7. Broadcast/Multicast Requirements 📣

Some existing industrial plant applications do not use broadcast or multicast addressing to communicate with field devices. Unicast address support is sufficient for them 💖

In some other industrial process automation environments, multicast over IP is used to deliver to multiple nodes that may be functionally similar or not. Example usages are:

  1. Delivery of alerts to multiple similar servers in an automation control room. Alerts are multicast to a group address based on the part of the automation process where the alerts arose (e.g., the multicast address "all-nodes-interested-in-alerts-for-process-unit-X"). This is always a restricted-scope multicast, not a broadcast 🚨
  2. Delivery of common packets to multiple routers over a backbone, where the packets result in each receiving router initiating multicast (sometimes as a full broadcast) within the LLN. For instance, this can be a byproduct of having potentially physically separated backbone routers that can inject messages into different portions of the same larger LLN 🌐
  3. Publication of measurement data to more than one subscriber. This feature is useful in some peer-to-peer control applications. For example, level position may be useful to a controller that operates the flow valve and also to the overfill alarm indicator. Both the controller and the alarm indicator would receive the same publication sent as a multicast by the level gauge 📊

All of these uses require a 1:N security mechanism as well 🔐 They aren't of any use if the end-to-end security is only point-to-point.

It is quite possible that first-generation wireless automation field networks can be adequately useful without either of these capabilities 🍼 But in the near future, wireless field devices with communication controllers and protocol stacks will require control and configuration, such as firmware downloading, that may benefit from broadcast or multicast addressing 🚀

The routing protocol SHOULD support multicast addressing. 🫶

8. Protocol Performance Requirements 🏎️

The routing protocol MUST converge after the addition of a new device within several minutes, and SHOULD converge within tens of seconds, such that a device is able to establish connectivity to any other point in the network or determine that there is a connectivity issue ⏱️ Any routing algorithm used to determine how to route packets in the network MUST be capable of routing packets to and from a newly added device within several minutes of its addition, and SHOULD be able to perform this function within tens of seconds.

The routing protocol MUST distribute sufficient information about link failures to enable traffic to be routed such that all service requirements (especially latency) continue to be met 🙌 This places a requirement on the speed of distribution and convergence of this information, as well as the responsiveness of any routing algorithms used to determine how to route packets. This requirement only applies at normal link failure rates (see Section 5) and MAY degrade during failure storms ⛈️

Any algorithm that computes routes for packets in the network MUST be able to perform route computations in advance of needing to use the route 🗺️ Since such algorithms are required to react to link failures, link usage information, and other dynamic link properties as the information is distributed by the routing protocol, the algorithms SHOULD recompute routes based on the receipt of new information 💅✨

9. Mobility Requirements ✨

okay so economic stuff has led to fewer trained workers in the plant 😭 and a super common storyline is the "wireless worker" 📱. this person carries a PDA or something similar and gets more done in less time than the older, better-trained workers they replace 💅. whether that premise holds up is a separate question, but the use case keeps coming up: the worker wirelessly connects to the plant IT system to download documentation, instructions, etc., and needs to connect "directly" to the sensors and control points in or near the equipment they're working on 🔧. this "direct" connection could possibly run over the normal LLNs data collection network 🌐. it's likely to need higher bandwidth and lower latency than the normal data collection operation, so it's giving high-performance energy 🔥

PDAs are typically the user interfaces for plant historians, asset management systems, and the like 🧚. it's still undecided whether these PDAs will use the LLN directly to talk to field sensors, or whether they'll use other wireless connectivity that proxies back into the field or to anywhere else 🤷‍♀️.

okay so the routing protocol SHOULD support the wireless worker with fast network connection times of a few of seconds ⏱️, and low command and response latencies to the plant behind the LLN access points, to applications, and to field devices 💖. the routing protocol SHOULD also support the bandwidth allocation for bulk transfers between the field device and the handheld device of the wireless worker 📦. and the routing protocol SHOULD support walking speeds for maintaining network connectivity as the handheld device changes position in the wireless network 🚶‍♀️✨.

some field devices will be mobile too 🚜. they might sit on moving parts such as rotating components, or they might be on vehicles such as cranes or fork lifts 🏗️. the routing protocol SHOULD support vehicular speeds of up to 35 kmph 🏎️💨. no cap, that's a lot of movement to keep up with 😤.

10. Manageability Requirements 🎀

the process and control industry is manpower constrained, and the aging demographics of plant personnel are creating a looming manpower problem across lots of markets 😩. the goal for industrial networks is that installation doesn't require any new skills for plant personnel 🙌. a person should install the wireless sensor or wireless actuator the same way they'd install the wired one, except the step to connect the wire is eliminated 💅.

most users actually want even more simplified provisioning, a plug and play operation that's totally transparent to them 🧚‍♀️✨. this needs open and untrusted side channels for new joiners, and it needs strong and automated authentication so networks can automatically accept or reject new joiners 🔐. ideally, for a user, adding new routing devices should be as easy as dragging and dropping an icon from a pool of authenticated new joiners into a pool for the wired domain that this new sensor should connect to 🖱️💖. under the hood, invisible to the user, auditable security mechanisms should handle new device authentication and secret join key distribution 🫶. these more sophisticated 'over the air' secure provisioning methods should get rid of the need for traditional configuration tools to set up devices before they're ready to securely join an LLN access point 🙅‍♀️.

the routing protocol SHOULD be fully configurable over the air as part of the joining process of a new routing device 📡.

there will be lots of new applications where, even without any human intervention at the plant, devices that have never been on site before should be allowed to connect anyway, based on their credentials and cryptographic capabilities 🔑. examples include third-party road tankers, rail cargo containers with overfill protection sensors, or consumer cars that need to be refueled with hydrogen by robots at future fueling stations 🚛🚗.

the routing protocol for LLNs is expected to be easy to deploy and manage 🌸. since the number of field devices in a network is large, provisioning them manually may not make sense 😵‍💫. the proper operation of the routing protocol MAY require that the node be commissioned with information about itself, like identity, security tokens, radio standards and frequencies, etc. 🧾.

the routing protocol SHOULD NOT require preprovisioning of information about the environment where the node will be deployed 🚫. the routing protocol MUST enable the full discovery and setup of the environment (available links, selected peers, reachable network) 🔍. and the protocol MUST enable the distribution of its own configuration to be performed by some external mechanism from a centralized management controller 🏢💜.

11. Antagonistic Requirements 🥊

this document has a bunch of strongly required constraints on the ROLL routing protocol 💪. some of those strong requirements might look antagonistic, and so seem impossible to fulfill at the same time 😬.

for instance, the strong requirement of power economy applies to general routing but is variable, because it's reasonable to spend more energy on keeping a short emergency closed-loop path available than on maintaining an alert path used for regular updates on the device's operating status ⚡. in the same way, the strong requirement on easy provisioning doesn't line up neatly with the strong security requirements that might be needed to implement a factory policy 🔐. still, a non-default, non-trivial setup can be acceptable as long as the default configuration lets a device join with some degree of security 🫶.

convergence time and network size are also antagonistic 🌍. the values expressed in Section 8 ("Protocol Performance Requirements") apply to an average network with tens of devices 📊. a backbone can keep that level of performance while still letting the network grow to thousands of nodes 🦋. in any case, it's acceptable for convergence time to grow reasonably with network size 📈.

12. Security Considerations 🛡️

okay this part is seriously important 😤. wireless sensor networks in industrial automation run systems with major financial and human safety implications, so security is a huge deal 🚨. levels of security violation that get tolerated as a "cost of doing business" in the banking industry are NOT acceptable here, when literally thousands of lives may be at risk 💀.

security gets confused with a guarantee of availability all the time 🙈. when talking about wireless security, it's important to clearly separate the risks of temporarily losing connectivity, say due to a thunderstorm ⛈️, from the risks of knowledgeable adversaries attacking a wireless system 🕵️‍♀️. the conscious attacks need to be split between 1) attacks on the actual application served by the wireless devices and 2) attacks that exploit the presence of a wireless access point that may provide connectivity onto legacy wired plant networks 🔓. the second kind has little to do with the wireless devices in the LLNs. and in that second type, access points that could be wireless backdoors, letting an attacker outside the fence reach typically unsecured process control and/or office networks, are typically the ones that create exposures where lives are at risk 😱. this implies the LLN access point on its own MUST possess functionality that guarantees domain segregation, and thus prohibits many types of traffic further upstream 🚧.

the current generation of industrial wireless device manufacturers specifies security at the MAC (Media Access Control) layer and the transport layer 🧱. a shared key is used to authenticate messages at the MAC layer 🗝️. at the transport layer, commands are encrypted with statistically unique, randomly generated end-to-end session keys 🎲. HART7 [IEC62591] and ISA100.11a are examples of security systems for industrial wireless networks 🌟.

symmetric key encryption and authentication mechanisms at the MAC and transport layers may protect reasonably well during the lifecycle 👍. however, the initial network boot (provisioning) step often needs more sophisticated steps to securely land the initial secret keys in field devices 🥺. also, it is vital that during these steps, ease of deployment and the freedom to mix and match products from different suppliers don't complicate life for the people deploying and commissioning 🫠. given the average skill levels in the field and the serious resource constraints in the market, investing a little bit more in sensor-node hardware and software, so new devices can automatically be deemed trustworthy and automatically join the domains they should join, with just one drag-and-drop action for those in charge of deploying, will yield faster adoption and spread of LLN technology 🚀💖.

industrial plants may not keep the same level of physical security for field devices that's associated with traditional network sites, like locked IT centers 🔒. in industrial plants, it MUST be assumed that field devices have marginal physical security and might be compromised 😰. the routing protocol SHOULD limit the risk incurred by one node being compromised, for instance by proposing a non-congruent path for a given route and balancing traffic across the network ⚖️.

the routing protocol SHOULD compartmentalize the trust placed in field devices, so that a compromised field device doesn't destroy the security of the whole network 🧩. the routing MUST be configured and managed using secure messages and protocols that prevent outsider attacks and limit insider attacks from field devices installed in insecure locations in the plant 🔐🏭.

the wireless environment typically forces us to abandon classical 'by perimeter' thinking when securing network domains 🌊. wireless nodes in LLN networks should therefore be regarded as little islands with trusted kernels, sitting in an ocean of untrusted connectivity, an ocean that might be full of pirate ships 🏴‍☠️🌊. because of that, confidence in node identity and the ability to challenge the authenticity of source node credentials matter a lot more 🫶. cryptographic boundaries inside devices that clearly mark the border between trusted and untrusted areas need to be drawn ✏️. protection against compromise of the cryptographic boundaries inside device hardware is outside the scope of this document 🙅‍♀️.

note that because nodes are usually expected to be capable of routing, the end-node security requirements are usually a superset of the router requirements 💡. that way, an end node can't be used to inject forged information into the network that could alter plant operations 🚫.

additional details of security across all application scenarios are provided in the ROLL security framework [ROLL-SEC-FMWK] 📚. the implications of these security requirements for the routing protocol itself are a topic for future work 🔮.

13. Acknowledgements 🫶

huge thanks to Rick Enns, Alexander Chernoguzov, and Chol Su Kang for their contributions 🌷✨.

14. References 📚

14.1. Normative References

  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.

14.2. Informative References ✨

okay so these are the bonus reading list 📚 not required to implement, but super helpful for the vibe and the background

  • [HART] HART (Highway Addressable Remote Transducer) Communication Foundation, "HART Communication Protocol and Foundation - Home Page", <http://www.hartcomm.org>. 🔧
  • [IEC61158] IEC, "Industrial communication networks - Fieldbus specifications", IEC 61158 series. 🏭
  • [IEC62591] IEC, "Industrial communication networks - Wireless communication network and communication profiles - WirelessHART", IEC 62591. 📡
  • [IEEE802.15.4] IEEE, "Telecommunications and information exchange between systems -- Local and metropolitan area networks -- Specific requirements Part 15.4: Wireless Medium Access Control (MAC) and Physical Layer (PHY) Specifications for Low-Rate Wireless Personal Area Networks (WPANs)", IEEE 802.15.4, 2006. 💜
  • [ISA100.11a] ISA, "Wireless systems for industrial automation: Process control and related applications", ISA 100.11a, May 2008, <http://www.isa.org/Community/SP100WirelessSystemsforAutomation>. 🌸
  • [RFC4919] Kushalnagar, N., Montenegro, G., and C. Schumacher, "IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs): Overview, Assumptions, Problem Statement, and Goals", RFC 4919, August 2007. 🦋
  • [ROLL-SEC-FMWK] Tsao, T., Alexander, R., Dohler, M., Daza, V., and A. Lozano, "A Security Framework for Routing over Low Power and Lossy Networks", Work in Progress, September 2009. 🔐
  • [ROLL-TERM] Vasseur, JP., "Terminology in Low power And Lossy Networks", Work in Progress, October 2009. 📖

Authors' Addresses 🎀

Here are the people who wrote it, bestie. Ily all 💖

Kris Pister (editor)
Dust Networks
30695 Huntwood Ave.
Hayward, CA  94544
USA

EMail: kpister@dustnetworks.com


Pascal Thubert (editor)
Cisco Systems
Village d'Entreprises Green Side
400, Avenue de Roumanille
Batiment T3
Biot - Sophia Antipolis  06410
FRANCE

Phone: +33 497 23 26 34
EMail: pthubert@cisco.com


Sicco Dwars
Shell Global Solutions International B.V.
Sir Winston Churchilllaan 299
Rijswijk  2288 DC
Netherlands

Phone: +31 70 447 2660
EMail: sicco.dwars@shell.com


Tom Phinney
Consultant
5012 W. Torrey Pines Circle
Glendale, AZ  85308-3221
USA

Phone: +1 602 938 3163
EMail: tom.phinney@cox.net

🫶✨

📦 where this page comes from