RFC 9550

Deterministic Networking (DetNet): Packet Ordering Function

📅 published March 2024, from ietf

Deterministic Networking (DetNet): Packet Ordering Function ✨

Okay bestie, this is RFC 9550 🎀 It's titled Deterministic Networking (DetNet): Packet Ordering Function, and it's an Informational doc (ISSN 2070-1721), so it's sharing knowledge rather than setting a standard 💅 It came out in March 2024. The editor is B. Varga, Ed., and the other authors are J. Farkas, S. Kehrer, and T. Heer. The affiliations are Ericsson and Belden. The header doesn't list anything it obsoletes or updates, so it's a standalone drop 🦋

Abstract 🌸

So the replication and elimination functions in the DetNet architecture can end up with packets arriving out of order, and for some time-sensitive apps that's a total no-go 😭 The Packet Ordering Function (POF) algorithms in this doc fix that by putting packets back in the right order when replication and elimination are used in DetNet networks. The POF only sorts things out within the latency bound of a DetNet flow, and it does not add any extra reliability. Know your limits, queen 👑

Status of This Memo 🧚

This doc is not an Internet Standards Track spec. It's published purely for information 💖

It's a product of the IETF, so it does reflect the consensus of the IETF community. It went through public review and got the okay from the Internet Engineering Steering Group (IESG). Not everything the IESG approves is eligible for any level of Internet Standard, which is explained in Section 2 of RFC 7841 🙈

Want the current status, any errata, or a way to send feedback? Head to https://www.rfc-editor.org/info/rfc9550 ✨

Copyright Notice 💜

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stood on the publication date. Please read them carefully, because they explain your rights and restrictions for this doc 📜 Code Components pulled out of this doc must include the Revised BSD License text described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Revised BSD License describes 🫶

Table of Contents 🍓

1.  Introduction
2.  Terminology
  2.1.  Terms Used in This Document
  2.2.  Abbreviations
3.  Requirements for POF Implementations
4.  POF Algorithms
  4.1.  Prerequisites and Assumptions
  4.2.  POF Building Blocks
  4.3.  The Basic POF Algorithm
  4.4.  The Advanced POF Algorithm
  4.5.  Further Enhancements of the POF Algorithms
  4.6.  Selecting and Using the POF Algorithms
5.  Control and Management Plane Parameters for POF
6.  Security Considerations
7.  IANA Considerations
8.  References
  8.1.  Normative References
  8.2.  Informative References
Acknowledgements
Authors' Addresses

1. Introduction ✨

Okay so [RFC8655] defines the Packet Replication Function (PRF) and Packet Elimination Function (PEF) in DetNet, and the whole point is getting extremely low packet loss 🦋 The PRF and PEF protect DetNet flows by sending copies of the same packet over several maximally disjoint paths, then using sequencing info to kill the duplicates. A possible implementation is described in [IEEE8021CB], and its YANG model is defined in [IEEEP8021CBcv] 💅

Here's the tea: per-packet replication and elimination can cause packets to be delivered out of order, and some deterministic apps just cannot handle that 😭 Correcting the order isn't simple, so this doc specifies a Packet Ordering Function (POF) in detail. [RFC8655] tells you what the POF's observable result is (packets get reordered), but it doesn't go into any implementation details 🌈

Traditionally, out-of-order delivery has been dealt with higher up the OSI stack, at the endpoints or hosts. Think the TCP stack when packets get handed to the application layer. It has basically never been handled inside the network, in nodes working at Layer 2 or Layer 3 of the OSI model 🙈

Figure 1 shows a DetNet flow where Packet Replication, Elimination, and Ordering Functions (PREOF) are applied as packets travel from source to destination 🧚

                                     +------------+
              +-----------E1----+    |            |
 +----+       |            |    +---R3---+        |          +----+
 |src |------R1        +---+             |        E3----O1---+ dst|
 +----+       |        |                 E2-------+          +----+
              +-------R2                 |
                       +-----------------+

R: replication point (PRF)
E: elimination point (PEF)
O: ordering function (POF)

            Figure 1: PREOF Scenario in a DetNet Network

In general, PREOF needs sequencing info included in the packets of a DetNet compound flow. You can do this by adding a sequence number as part of the DetNet encapsulation [RFC8655] 🎀 The sequencing info is normally added once, at or close to the source 💖

Different apps react to out-of-order delivery in totally different ways 😭 One packet arriving out of order (say, packet order #1, #3, #2, #4, #5) might count as a single error for one app, but another app sees it as three errors in a row 💀 In industrial settings, three errors in a row is a typical error threshold, and it can make the app stop (for example, by going into a fail-safe state) 🚨

The POF makes sure packets get delivered in order within the latency bound of the DetNet flow. It does not fix errors in the packet flow, like duplicate packets or packets that show up too late 🙅‍♀️

2. Terminology 🎀

2.1. Terms Used in This Document 💜

This doc uses the terminology from the DetNet architecture [RFC8655], and it assumes you already know that doc and its terms 📚

2.2. Abbreviations 🌸

These are the abbreviations used in this doc 💅

DetNet   Deterministic Networking

PEF      Packet Elimination Function

POF      Packet Ordering Function

PREOF    Packet Replication, Elimination, and Ordering Functions

PRF      Packet Replication Function

3. Requirements for POF Implementations 🦋

These are the requirements for POF implementations:

To solve the out-of-order delivery problem of the replication and elimination functions of DetNet networks.
To consider the delay bound requirement of a DetNet flow.
To be simple and to require only a minimum set of states, configuration parameters, and resources per DetNet flow in network nodes.
To add minimal or no delay to the forwarding process of packets.
To not require synchronization between PREOF nodes.

Some aspects are explicitly out of scope for a POF 🙅‍♀️

To eliminate the delay variation caused by the packet ordering. Dealing with delay variation is a DetNet forwarding sub-layer target, and it can be achieved, for example, by placing a de-jitter buffer or flow regulator (e.g., shaping) function after the POF.

4. POF Algorithms ✨

4.1. Prerequisites and Assumptions 🧚

The POF algorithms in this doc make some assumptions and trade-offs about what the sequence of received packets looks like 🫶 Specifically, they assume a PEF runs on the incoming packets before they get handed to the POF. So the incoming sequence can be out of order or incomplete, but it cannot contain duplicate packets. The PREOF pieces run independently, with no state exchange needed between the PEF and the POF, or between the PRF and the POF. If duplicate packets do get presented to the POF anyway, that's an error case that can lead to out-of-order delivery of the duplicates and to increased delays 😭

The algorithms also need the delay difference between two replicated packets that reach the PEF before the POF to be bounded and known. Error cases that break this condition (for example, a packet arriving later than the bound) will end up with out-of-order packets 💀

There are some trade-offs baked in too. For simplicity, the algorithms allow some out-of-order packets right after initialization. If that's not acceptable for you, Section 4.5 offers an alternative initialization scheme that prevents out-of-order packets during the initialization phase 💖

4.2. POF Building Blocks 🌈

This doc describes a POF for DetNet networks. The POF's configuration parameters can be derived when you engineer the DetNet flow through the network ✨

The POF method is provided through the following pieces:

Delay calculator: : Calculates buffering time for out-of-order packets. Buffering time considers (i) the delay difference of paths used for forwarding the replicated packets and (ii) the bounded delay requirement of the given DetNet flow.

Conditional delay buffer: : Used for buffering the out-of-order packets of a DetNet flow for a given time.

Note: The conditional delay buffer of the POF increases the burstiness of the traffic, because it only adds delay for some of the packets. Watch out for that 👀

Figure 2 shows the building blocks of a possible POF implementation 🦋

                  +------------+        +--------------+
                  | Delay calc |        | Conditional  |
               +--| for packet >--->>---| Delay Buffer >--+
               |  +------------+        +--------------+  |
               |                                          |
        +------^--------+                                 |
   ->>--| POF selector  >---------------------------------+-->>----
        | (Flow ident.) |
        +---------------+

   ->>- packet flow

                     Figure 2: POF Building Blocks

4.3. The Basic POF Algorithm ✨

okay bestie, the basic POF algorithm is all about patience 🧚💖 It holds back every out-of-order packet until the earlier ones show up, or until a given time (POFMaxDelay) runs out. no cap, it's giving very thoughtful friend energy.

Here's how it works, step by step 💅

The sequence number of the last forwarded packet (POFLastSent) gets stored for each DetNet flow 🎀
The sequence number (seq_num) of a received packet gets compared to the one stored in POFLastSent 👀
If (seq_num <= POFLastSent + 1):
- Then the packet gets forwarded without buffering, and POFLastSent is updated (POFLastSent = seq_num) 🦋
- Else, the received packet gets buffered 🫶
A buffered packet gets forwarded from the buffer when its seq_num becomes equal to POFLastSent + 1, OR when the predefined time (POFMaxDelay) elapses ⏰
When a packet gets forwarded from the buffer, POFLastSent is updated with its seq_num (POFLastSent = seq_num) ✨

Notes, because the details matter 📝

The gap between sequence numbers in consecutive packets is bounded thanks to the history window of the elimination function that sits before the POF 🌸 Which means <= still works even though the sequence number space is circular. A possible implementation of the PEF and how the history window plays into it are described in [IEEE8021CB]. iykyk 🙈
The basic POF algorithm can be extended to handle multiple failure scenarios (so simultaneous packet loss and out-of-order packets). This happens when the timer expiring for a packet with sequence number N triggers the release of some packets with a sequence number smaller than N 🧠

The state the basic POF algorithm uses (that's POFLastSent) needs initialization and maintenance. The vibe goes like this 🌈

The next received packet gets forwarded, and POFLastSent gets updated, when the POF is reset OR when no packet is received for a predefined time (POFTakeAnyTime) 💜
Resetting the POF wipes out all packets from the time-based buffer the POF uses 🧹

The basic POF algorithm has two parameters to engineer 💖

POFMaxDelay, which cannot be smaller than the delay difference of the paths used by the flow 🛣️
POFTakeAnyTime, which gets calculated from several factors. For example: the settings of the elimination function(s) relating to RECOVERY_TIMEOUT before the POF, the flow characteristics (like inter-packet time), and the delay difference of the paths used by the flow 🧮

Designing these parameters is out of scope for this document. not me saying "that's a problem for another day" 🙈

Note: multiple network failures can impact the POF (for example, a complete outage of all redundant paths) 💀

The basic POF algorithm increases the delay of packets by up to POFMaxDelay time. In-order packets are not delayed, which is so iconic 👑 This basic POF method can be applied in every network scenario where the remaining delay budget of a flow at the POF point is larger than POFMaxDelay time.

Figure 3 shows the delay budget situation at the POF point 🎀

                             Path delay
                             difference
                           /-------------/
   <- path with min delay ->             /-- remaining delay budget --/

   |-----------------------|-------------|----------------------------|
   0                       t1            t2                           T

   <-------- path with max delay -------->

   /-------------------- delay budget at POF point -------------------/

             Figure 3: Delay Budget Situation at the POF Point

4.4. The Advanced POF Algorithm 💅

so when the remaining delay budget of a flow at the POF point is smaller than POFMaxDelay time, the basic method needs some upgrades 🔥

The tea is that packets on the longest path can't be buffered if you want to keep the flow's delay budget. Also, a packet forwarded over the longest path needs no buffering, because it's the last chance to deliver a packet with that sequence number. All the replicas already arrived via the shorter path(s) 🫶

The advanced POF algorithm requires extensions to the basic POF algorithm 🧩

to identify the received packet's path at the POF location, and
to make the value of POFMaxDelay for buffered packets path dependent (POFMaxDelay_i, where i notes the path the packet has used).

So the advanced POF algorithm figures out the path of a given packet and uses that info to pick the predefined time (POFMaxDelay_i) for the buffered packet. In the advanced POF algorithm, POFMaxDelay becomes an array with a predefined, path-specific buffering time for each redundant path of a flow 🌸 The values in the POFMaxDelay array are engineered to fulfill the delay budget requirement. chef's kiss 👌

Designing these parameters is out of scope for this document 🙈

Note: for the advanced POF algorithm, path-dependent delays might cause multiple packets to hit "maximum delay reached" at exactly the same time. Their transmission order should be done in seq_num order 🦋

The method for identifying a packet's path at the POF location depends on the network scenario. It can be done with various techniques, for example: using ingress interface information, or encoding the path in the packet itself (like replication functions setting a different FlowID per member flow at their egress, with that FlowID then used to identify the path at the POF). Or other means 🧚 Methods for identifying the packet's path are out of scope for this document.

Note: when using the advanced POF algorithm, it might be advantageous to combine PEF and POF locations in the DetNet network. That can simplify the method used for identifying the packet's path at the POF location 💜

4.5. Further Enhancements of the POF Algorithms 🌈

POF algorithms can be leveled up by telling apart the initialization case from normal operation. The cost is more state and a more sophisticated implementation, but it's worth it 💖 These enhancements could, for example, react better after some failure scenarios (like a complete outage of all paths of a DetNet flow), and they can depend on the PEF implementation.

The tricky part of POF initialization is that it's not known whether the first received packet is in-order or out-of-order (for example, after a reset). The original initialization (see Section 4.3) considers the first packet as in-order. So out-of-order packets arriving during the "POFMaxTime"/"POFMaxTime_path_i" time, after the first packet is received, cannot be corrected 😭 The motivation behind that setup is simplicity of POF implementation.

A possible enhancement of POF initialization goes like this 🎀

After a reset, all received packets get buffered with their predefined timer (POFMaxTime/POFMaxTime_path_i).
No basic or advanced POF rules are applied until the first timer expires.
When the first timer expires, the packet with the lowest seq_num in the buffer gets selected and forwarded, and POFLastSent is set with its seq_num.
The basic or advanced POF rules get applied to the packet(s) in the buffer and to the packets received afterwards.

4.6. Selecting and Using the POF Algorithms 🧠

choosing a POF algorithm depends on the network scenario and the remaining delay budget of a flow. Using these algorithms and calculating their parameters needs proper design 🛠️ Knowing the path delay difference is essential for the POF algorithms described here. Failure scenarios that break the design assumptions can impact the POF result. For example, a packet received outside the expected worst-case delay window (calculated from the path delay difference) can cause unwanted out-of-order delivery 😬

In DetNet scenarios, there is always an elimination function before the POF, so duplicates are not considered by the POF 💅 Putting them together in the same node lets the POF take PEF events and states into account during reordering. For example, under normal circumstances the gap between sequence numbers in consecutive packets is bounded by the history window of the PEF. But in some scenarios (like a reset of the sequence number), that gap can be much larger than the size of the history window 🙃

5. Control and Management Plane Parameters for POF 🎛️

POF algorithms need the following parameters set 💖

Basic POF
- POFMaxDelay
- POFTakeAnyTime
Advanced POF
- POFMaxDelay_i for each possible path i
- POFTakeAnyTime
- Configuration(s) related to network path identification

Note: in a proper design, POFTakeAnyTime is always larger than POFMaxDelay 🌸

6. Security Considerations 🔐

okay this part is serious, bestie 😱 PREOF-related security considerations (including POF) are described in Section 3.3 of [RFC9055]. There are no additional POF-related security considerations originating from this document. go read Section 3.3 of [RFC9055] before you do anything else 💀

7. IANA Considerations 📚

This document has no IANA actions. nothing to register here, periodt 💅

8. References 📖

8.1. Normative References

[RFC8655]  Finn, N., Thubert, P., Varga, B., and J. Farkas,
           "Deterministic Networking Architecture", RFC 8655,
           DOI 10.17487/RFC8655, October 2019,
           <https://www.rfc-editor.org/info/rfc8655>.

[RFC9055]  Grossman, E., Ed., Mizrahi, T., and A. Hacker,
           "Deterministic Networking (DetNet) Security
           Considerations", RFC 9055, DOI 10.17487/RFC9055, June
           2021, <https://www.rfc-editor.org/info/rfc9055>.

8.2. Informative References

[IEEE8021CB]
           IEEE, "IEEE Standard for Local and metropolitan area
           networks -- Frame Replication and Elimination for
           Reliability", IEEE Std 802.1CB-2017,
           DOI 10.1109/IEEESTD.2017.8091139, October 2017,
           <https://standards.ieee.org/standard/802_1CB-2017.html>.

[IEEEP8021CBcv]
           IEEE, "IEEE Standard for Local and metropolitan area
           networks -- Frame Replication and Elimination for
           Reliability - Amendment 1: Information Model, YANG Data
           Model, and Management Information Base Module", IEEE Std
           802.1CBcv-2001, DOI 10.1109/IEEESTD.2022.9715061, February
           2022, <https://standards.ieee.org/ieee/802.1CBcv/7285/>.

Acknowledgements 🫶

The authors want to give a huge shoutout to Gyorgy Miklos, Ehsan Mohammadpour, Ludovic Thomas, Greg Mirsky, Jeong-dong Ryoo, Fan Yang, Toerless Eckert, Norman Finn, and Ethan Grossman 💖 for their insightful comments and productive discussion, which helped make the document better. the main character supporting cast 👑

Authors' Addresses 🏡

Balazs Varga (editor)
Ericsson
Budapest
Magyar Tudosok krt. 11.
1117
Hungary
Email: balazs.a.varga@ericsson.com

Janos Farkas
Ericsson
Budapest
Magyar Tudosok krt. 11.
1117
Hungary
Email: janos.farkas@ericsson.com

Stephan Kehrer
Belden Electronics GmbH
Stuttgarter Strasse 45-51.
72654 Neckartenzlingen
Germany
Email: Stephan.Kehrer@belden.com

Tobias Heer
Belden Electronics GmbH
Stuttgarter Strasse 45-51.
72654 Neckartenzlingen
Germany
Email: Tobias.Heer@belden.com

📦 where this page comes from

ietf: rfc9550 2024-03
💖 girlified by claude-haiku-5-5 on 2026-10-19: a language model's rewrite of the original, under its terms