RFC 8557

Deterministic Networking Problem Statement

📅 published May 2019, from ietf

Deterministic Networking Problem Statement 💖

Okay bestie, let's get into RFC 8557, a 2019 Informational doc from Norman Finn (Huawei Technologies Co. Ltd) and Pascal Thubert (Cisco) 🦋. It's Informational, not a Standards Track spec, and it's published under ISSN 2070-1721. The header doesn't list anything it obsoletes or updates, so it's standing on its own ✨.

Abstract

This paper is all about the needs in lots of different industries for setting up multi-hop paths for specific flows that have deterministic properties 🌸.

Status of This Memo

Real talk: this doc is not an Internet Standards Track spec. It's just being published for info 📚.

It's a product of the IETF 💅, which means it reflects what the IETF community agrees on. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG) 🎀. Not every IESG-approved doc is up for any level of Internet Standard, though. For that, check Section 2 of RFC 7841 🙈.

If you want the latest status, errata, or a place to send feedback, head to https://www.rfc-editor.org/info/rfc8557 🧚.

Copyright (c) 2019 IETF Trust and the people listed as the document's authors 💜. All rights reserved.

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info), which were in effect when it was published 📜. Read these carefully, because they spell out your rights and restrictions for this doc 👀. Any Code Components pulled out of this doc must include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License ⭐.

Table of Contents

   1. Introduction ....................................................2
   2. On Deterministic Networking .....................................4
   3. Problem Statement ...............................................6
      3.1. Supported Topologies .......................................6
      3.2. Flow Characterization ......................................6
      3.3. Centralized Path Computation and Installation ..............7
      3.4. Distributed Path Setup .....................................8
      3.5. Duplicated Data Format .....................................8
   4. Security Considerations .........................................9
   5. IANA Considerations .............................................9
   6. Informative References .........................................10
   Acknowledgments ...................................................11
   Authors' Addresses ................................................11

1. Introduction ✨

Okay so, [RFC8578] shows that it's not just Industrial Automation and Control Systems (IACSs) that need this stuff 🏭. Lots of industries have strong, pretty similar needs for deterministic network services with latency guarantees and ultra-low packet loss 💀. Literally iconic 👑.

Because everyone wants this, the IEEE 802.1 Audio Video Bridging (AVB) Task Group grew up into the Time-Sensitive Networking (TSN) [IEEE-802.1TSNTG] Task Group (TG) 🦋. Its crowd got way bigger too, with folks from the industrial and vehicular markets joining in 🚗.

With that expansion, the networks we're talking about are getting bigger and more structured 🌈. They need deterministic forwarding way past the LAN boundary. For example, an IACS splits its network along the broad lines of the Purdue Enterprise Reference Architecture (PERA) [ISA95] 🏗️. It usually uses deterministic LANs for Purdue level 2 control systems. Meanwhile, public infrastructure like electricity automation needs deterministic properties over the wide area ⚡. Implementers have realized that when IT and Operation Technology (OT) networks come together, you need Layer 3 as well as Layer 2 capabilities 🫶.

The initial user base has been almost totally focused on Ethernet physical media and Ethernet-based bridging protocols from several Standards Development Organizations (SDOs) 🔌. But the Layer 3 need we just described must not be limited to Ethernet and Ethernet-like media 🙅‍♀️. Such media absolutely must be covered by any useful Deterministic Networking (DetNet) architecture 🔥. And the cooperation between the IETF and other SDOs must not be limited to the IEEE or the IEEE 802 organizations 🌸. Finally, the completed and ongoing work in other SDOs, especially IEEE 802, is an obvious starting point for a DetNet architecture 💅. Even so, we must not assume that this other work boxes in the space where the DetNet architecture can grow 🦋.

Deterministic networks have specific requirements for using routed networks to support these apps. So a new model has to be proposed to bring this determinism into IT implementations 🧚. The proposed model should enable a fully scheduled operation, orchestrated by a central controller 🎀. It may also support a more distributed operation with (probably lesser) capabilities ✨. Either way, the model should not compromise the network's ability to keep carrying the kinds of traffic it already carries today, alongside the new, more deterministic flows 💖. Quick note: "Deterministic Networking Architecture" [DetNet-Arch] is the doc the DetNet Working Group made to describe that model 📚.

At the time of writing, here's what's expected 👀:

  • once the abstract model is agreed upon, the IETF will specify
    • (1) the signaling elements to be used to establish a path, and
    • (2) the tagging elements to be used to identify the flows that are to be forwarded along that path
  • the IETF will specify the necessary protocols or protocol additions, based on relevant IETF technologies, to implement the selected model 🌈

The dream outcome here is being able to set up a multi-hop path over an IP or MPLS network for a particular flow, with given timing and precise throughput requirements 🍓. Then you carry that flow along the multi-hop path with characteristics like low latency and ultra-low jitter 💜. You also get reordering and/or replication and elimination of packets over non-congruent paths for a higher delivery ratio, and/or zero congestion loss 🔥. And it all works no matter how many other flows are in the network 👑.

Depending on what the network can do and what state it's in, requests to set up a path from an end node or a network management entity may be granted or rejected 😬. An existing path may be moved or removed 🙈. And DetNet flows that go over their contract may face packet declassification and drop 💀.

2. On Deterministic Networking ✨

okay so the Internet isn't the only digital network that has blown up over the last 30-40 years 📈 video and audio entertainment, plus the control systems for machines, factories, and vehicles, are all everywhere now and basically run on digital tech. over the past 10 years, engineers in these fields realized they could save a lot of money and grow way faster by putting all of these different digital technologies onto packet networks 🚀 literally the glow-up of the decade 💅

so the goals of Deterministic Networking are two things 🎀 first, let apps with critical timing and reliability needs move off their special-purpose fieldbus tech (HDMI, CAN bus, PROFIBUS [PROFIBUS], and so on... even RS-232!! 💀) and onto packet tech in general and IP in particular. second, support those new apps AND existing packet network apps over the same physical network 🫶 in other words, a deterministic network is backwards compatible, meaning it can still carry statistically multiplexed traffic while keeping the properties of the deterministic flows it already accepted. ✨ no one gets left behind!!

[RFC8578] says apps in lots of different fields need some or all of a suite of features, and it's giving a big list 👑

  1. Time synchronization of all host and network nodes (routers and/or bridges), accurate to something between 10 nanoseconds and 10 microseconds, depending on the app. ⏱️
  2. Support for deterministic packet flows that:
    • Can be unicast or multicast. 🦋
    • Need absolute guarantees of minimum and maximum latency end to end across the network. Sometimes a tight jitter is required as well 😭
    • Need a packet loss ratio beyond the classical range for a particular medium, in the range of 10^-9 to 10^-12 or better on Ethernet, and on the order of 10^-5 in wireless sensor mesh networks. 🥹
    • Can, in total, absorb more than half of the network's available bandwidth. (Basically, massive over-provisioning is ruled out as a solution. not it 🙅‍♀️)
    • Cannot suffer throttling, congestion feedback, or any other network-imposed transmission delay, although the flows can be meaningfully characterized by either (1) a fixed, repeating transmission schedule or (2) a maximum bandwidth and packet size. 🧚
  3. Multiple methods for scheduling, shaping, limiting, and otherwise controlling the transmission of critical packets at each hop through the network data plane. 🎛️
  4. Robust defenses against misbehaving hosts, routers, or bridges, in both the data plane and the control plane. The guarantee is that a critical flow within its guaranteed resources cannot be affected by other flows, whatever the pressures on the network 🛡️ for more on the specific threats against DetNet, check "Deterministic Networking (DetNet) Security Considerations" [DetNet-Security]. this part is so important ngl 😤
  5. One or more methods for reserving resources in bridges and routers to carry these flows. 📦

time-synchronization techniques don't need to be handled by an IETF working group, bestie 🙌 there are already a bunch of standards for this, including IEEE 1588 [IEEE-1588], IEEE 802.1AS [IEEE-8021AS], and more 🌈

the needs around multicast, latency, loss ratio, and throttling avoidance exist because the algorithms the apps use demand them 🧠 they aren't just a copy-paste of fieldbus needs into a packet-based fieldbus simulation. they also reflect the fundamental math of controlling a physical system 🔥

with classical forwarding of latency-sensitive and loss-sensitive packets across a network, the interactions between different critical flows introduce fundamental uncertainties in delivery schedules 😵‍💫 the details of the queuing, shaping, and scheduling algorithms each bridge or router uses to control the output sequence on a given port change the detailed makeup of the output stream. for example, how finely one flow's packets are mixed in among those of other flows 🍓

and that has a strong effect on the buffer requirements, which in turn affects the latency guarantees that can be delivered by the next bridge or router along the path 🫠 that's why the IEEE 802.1 TSN TG defined a new set of queuing, shaping, and scheduling algorithms. with those, each bridge or router can compute the exact number of buffers to allocate for each flow or class of flows 💜

networking protocols commonly need robustness, and it matters extra in real-time control networks 🚨 expensive equipment, and even lives, can be lost due to misbehaving equipment. so yeah, robustness is not optional there 😭

reserving resources before packet transmission is the one fundamental shift in how network apps behave that you just can't avoid 💅 for starters, a network cannot deliver finite latency and practically zero packet loss to an arbitrarily high offered load. second, getting practically zero packet loss for unthrottled (though bandwidth-limited) flows means bridges and routers have to dedicate buffer resources to specific flows or classes of flows. so the requirements of each reservation have to be translated into the parameters that control each host's, bridge's, and router's queuing, shaping, and scheduling functions, and then delivered to those hosts, bridges, and routers 🎀

3. Problem Statement 🔍

3.1. Supported Topologies

sometimes the end point that runs the app is involved in the Deterministic Networking operation, like when it controls certain parts of its own throughput, such as the rate or the precise time it emits packets 🎯 in that case, the deterministic path runs end to end from app host to app host.

on the other hand, the deterministic part of a path might be a tunnel between an ingress point and an egress router 🚇 either way, routers and switches in between shouldn't need to know whether the path is end to end or a tunnel. ✨ it's giving discretion 🤫

it's clear DetNet isn't trying to set up deterministic paths over the global Internet 🌍 but there's still some confusion about how big a domain can be before a deterministic path can be set up in it 🤔 those limits may depend on the technology used to set the path up, and on whether that setup is centralized or distributed.

3.2. Flow Characterization

deterministic forwarding can only apply to flows with well-defined characteristics, like periodicity and burstiness 📊 before a path can be established to serve them, those characteristics have to be expressed, and the way the network can serve them (for instance, in shaping and forwarding operations) has to be specified too. no shortcuts, babe 💅

3.3. Centralized Path Computation and Installation

a centralized routing model, like the one provided with a Path Computation Element (PCE) (see [RFC4655]), enables global and per-flow optimizations 👑 this kind of model is very attractive, but a few issues are still unsolved. in particular:

  • whether and how the path computation can be installed by
    • an end device, or
    • a network management entity, and
  • how the path is set up, either
    • by installing state at each hop, with a direct interaction between the forwarding device and the PCE, or
    • along a path, by injecting a source-routed request at one end of the path, following classical Traffic Engineering (TE) models 🛤️

to enable a centralized model, DetNet should produce a description of the high-level interaction and data models to:

  • report the topology and device capabilities to the central controller 🗺️
  • establish a direct interface between the centralized PCE and each device under its control, in order to enable vertical signaling 📡
  • request a path setup for a new flow with particular characteristics over the service interface, and control it through its life cycle 🔄
  • provide support for life-cycle management for a path (instantiate/modify/update/delete) 🛠️
  • provide support for adaptability, to cope with various events like loss of a link 🩹
  • expose the status of the path to the end devices (User-Network Interfaces (UNIs)) 👀
  • provide additional reliability through redundancy, particularly with Packet Replication, Elimination, and Ordering Functions (PREOF). redundant paths may deliver packets out of order, and PREOF may need to correct the ordering 🧩
  • indicate the flows and packet sequences in-band with the flows. this is needed for flows that require PREOF in order to isolate duplicates and reorder packets at the end of the sequence 🎀

3.4. Distributed Path Setup

whether a distributed alternative without a PCE can be valuable could be studied as well 🧐 that kind of alternative could, for instance, build on Resource Reservation Protocol - TE (RSVP-TE) flows [RFC3209]. but the focus of the work should be delivering the centralized approach first 🫶

to get functionality similar to RSVP-TE, these steps would take place:

  1. Neighbors and their capabilities would be discovered and exposed, so a path can be computed that fits the DetNet constraints. Those are typically latency, time precision, and resource availability. 🔭
  2. A constrained path would be calculated with an improved version of Constrained Shortest Path First (CSPF) that is aware of DetNet. 🧮
  3. The path may be installed using a control protocol such as RSVP-TE, extended to enable flow identification, install new per-hop behavior such as Packet Replication, Elimination, and Ordering, and reserve physical resources for the flow. In that case, traffic flows could be transported through an MPLS-TE tunnel, using the resources reserved for this flow at each hop. 🚀

3.5. Duplicated Data Format

in some cases, duplicating and eliminating packets over non-congruent paths is required to reach a high enough delivery ratio for what the apps need 🎯 in these cases, a small number of packet formats and supporting protocols are required (preferably just one of each) to serialize the packets of a DetNet stream at one point in the network, replicate them at one or more points, and discard duplicates at one or more other points, possibly including the destination host. 🧃 using an existing solution would be preferable to inventing a new one, so why reinvent the wheel? 🙄💅

4. Security Considerations ✨

okay bestie, security in Deterministic Networking has a whole extra dimension 😭🔥 the timing of when a packet arrives can be just as important as what's inside it!! 🕰️💅 a man-in-the-middle attacker can add extra delays into a link and then keep adjusting them, which can mess up or sabotage a real-time app without ever cracking any encryption 💀 if you want a deep dive on this in a related context, check out [RFC7384] 🦋

so today, control networks mostly rely on being totally physically isolated to keep rogue access out 🔒 DetNet lets those networks be virtualized over a converged IT/OT infrastructure 🌈 but here's the catch: now flows can interact and interfere with each other because they share physical stuff like Ethernet trunks and radio spectrum 😬 the requirement is that there is no possible data leak from or into a deterministic flow 🫶 more generally, there must be no possible influence whatsoever from the outside on a deterministic flow 💜 the expectation is that physical resources are effectively tied to a given flow at a given point in time ⏱️ in that model, time-sharing of physical resources is totally transparent to each flow, because the flows have no clue whether other flows are using those resources at other times 👑

the overall security of a deterministic system has to cover:

  • the protection of the signaling protocol 📡
  • the authentication and authorization of the controlling nodes, including plug-and-play participating end systems 🔐
  • the identification and shaping of the flows 🎀
  • the isolation of flows from leakage and other influences from any activity sharing physical resources 🧚

the specific threats against DetNet are discussed further in [DetNet-Security] 💖

5. IANA Considerations 🌸

this document has no IANA actions, so nothing to register, bestie ✨

6. Informative References 📚

[DetNet-Arch] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", Work in Progress, draft-ietf-detnet-architecture-13, May 2019.

[DetNet-Security] Mizrahi, T., Grossman, E., Ed., Hacker, A., Das, S., Dowdell, J., Austad, H., Stanton, K., and N. Finn, "Deterministic Networking (DetNet) Security Considerations", Work in Progress, draft-ietf-detnet-security-04, March 2019.

[IEEE-1588] IEEE, "IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems", IEEE Standard 1588-2008, <https://standards.ieee.org/ findstds/standard/1588-2008.html>.

[IEEE-802.1TSNTG] IEEE Standards Association, "IEEE 802.1 Time-Sensitive Networking Task Group", <http://www.ieee802.org/1/pages/avbridges.html>.

[IEEE-8021AS] IEEE, "IEEE Standard for Local and Metropolitan Area Networks - Timing and Synchronization for Time-Sensitive Applications in Bridged Local Area Networks", IEEE 802.1AS-2011, <http://www.ieee802.org/1/pages/802.1as.html>.

[ISA95] ANSI/ISA, "Enterprise-Control System Integration - Part 1: Models and Terminology", <https://www.isa.org/isa95/>.

[PROFIBUS] IEC, "PROFIBUS Standard - DP Specification (IEC 61158 Type 3)", <https://www.profibus.com/>.

[RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, <https://www.rfc-editor.org/info/rfc3209>.

[RFC4655] Farrel, A., Vasseur, J.-P., and J. Ash, "A Path Computation Element (PCE)-Based Architecture", RFC 4655, DOI 10.17487/RFC4655, August 2006, <https://www.rfc-editor.org/info/rfc4655>.

[RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, October 2014, <https://www.rfc-editor.org/info/rfc7384>.

[RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", RFC 8578, DOI 10.17487/RFC8578, May 2019, <https://www.rfc-editor.org/info/rfc8578>.

Acknowledgments 🙏

the authors wish to thank Lou Berger, Pat Thaler, Jouni Korhonen, Janos Farkas, Stewart Bryant, Andrew Malis, Ethan Grossman, Patrick Wetterwald, Subha Dhesikan, Matthew Miller, Erik Nordmark, George Swallow, Rodney Cummings, Ines Robles, Shwetha Bhandari, Rudy Klecka, Anca Zamfir, David Black, Thomas Watteyne, Shitanshu Shah, Kiran Makhijani, Craig Gunther, Warren Kumari, Wilfried Steiner, Marcel Kiessling, Karl Weber, Alissa Cooper, and Benjamin Kaduk for their various contributions to this work 🫶💖

Authors' Addresses 📮

Norman Finn Huawei Technologies Co. Ltd 3755 Avocado Blvd. PMB 436 La Mesa, California 91941 United States of America

Phone: +1 925 980 6430 Email: norman.finn@mail01.huawei.com

Pascal Thubert Cisco Systems, Inc. Building D, 45 Allee des Ormes - BP1200 Mougins - Sophia Antipolis 06254 France

Phone: +33 4 97 23 26 34 Email: pthubert@cisco.com

📦 where this page comes from