RFC 9341

Alternate-Marking Method

📅 published December 2022, from ietf

RFC 9341: Alternate-Marking Method ✨

Bestie, this is RFC 9341 🦋 It's the Alternate-Marking Method spec for measuring packet loss, delay, and jitter on live traffic, and it obsoletes RFC 8321 💅 It's a Standards Track doc, published in December 2022, with ISSN 2070-1721. The editor is G. Fioccola (Huawei Technologies), with co-authors M. Cociglio (Telecom Italia), G. Mirsky (Ericsson), T. Mizrahi (Huawei Technologies), and T. Zhou (Huawei Technologies). Iconic team 👑

Status of This Memo ✨

This is an Internet Standards Track document 🎀

This document is a product of the Internet Engineering Task Force (IETF) 🌸 It represents the consensus of the IETF community, received public review, and was approved for publication by the Internet Engineering Steering Group (IESG). For more on Internet Standards, check out Section 2 of RFC 7841 🧚

Info about the current status of this doc, any errata, and how to send feedback lives at https://www.rfc-editor.org/info/rfc9341 💜

Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved 🔒

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the date of publication. Please actually read these, bestie, because they spell out your rights and restrictions for this doc 📜 Code Components pulled from this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Revised BSD License ⚠️

Table of Contents 🗺️

1.  Introduction
  1.1.  Summary of Changes from RFC 8321
  1.2.  Requirements Language
2.  Overview of the Method
3.  Detailed Description of the Method
  3.1.  Packet-Loss Measurement
  3.2.  One-Way Delay Measurement
    3.2.1.  Single-Marking Methodology
    3.2.2.  Double-Marking Methodology
  3.3.  Delay Variation Measurement
4.  Alternate-Marking Functions
  4.1.  Marking the Packets
  4.2.  Counting and Timestamping Packets
  4.3.  Data Collection and Correlation
5.  Synchronization and Timing
6.  Packet Fragmentation
7.  Recommendations for Deployment
  7.1.  Controlled Domain Requirement
8.  Compliance with Guidelines from RFC 6390
9.  IANA Considerations
10. Security Considerations
11. References
  11.1.  Normative References
  11.2.  Informative References
Acknowledgements
Contributors
Authors' Addresses

1. Introduction 🫶

Most Service Providers' networks carry traffic whose contents are super sensitive to packet loss [RFC7680], delay [RFC7679], and jitter [RFC3393] 😭

So we need tools and methods to monitor and measure network performance accurately, so providers can keep an eye on the quality of experience end customers are actually getting ✨ Performance monitoring is also useful for improving network management, like isolating network problems and troubleshooting 🔍

[RFC7799] defines Active, Passive, and Hybrid Methods of Measurement 💅 Active Methods depend on a dedicated measurement packet stream. Passive Methods are based only on observing an undisturbed, unmodified packet stream of interest. Hybrid Methods use a mix of Active and Passive Methods 🌸

This document proposes a performance monitoring technique called the "Alternate-Marking Method" 🦋 It's potentially usable with any kind of packet-based traffic, whether point-to-point unicast or multicast, including Ethernet, IP, and MPLS. The method mainly targets packet-loss measurement, but it can be easily extended to one-way or two-way delay and delay variation measurements too 🎀

The Alternate-Marking methodology in this document lets measurements taken at different points stay synchronized, by dividing the packet flow into batches. That way you get coherent counters and timestamps in every marking period, and you can measure the Performance Metrics for the monitored flow 💖

The method was explicitly designed for Passive or Hybrid measurements, as stated in [RFC8321]. But under the definitions in [RFC7799], the Alternate-Marking Method can be classified as Hybrid Type I 🧚 The reason: Alternate Marking can be implemented using reserved bits in the protocol header, and changing the value of those marking bits at the domain edges (not along the path) is formally considered a modification of the stream of interest 👑

Quick heads-up: this is a methodology document, so the mechanism used to transmit the counters and timestamps is out of scope here. More details on where Alternate-Marking applies are in [IEEE-NETWORK-PNPM] 📚

1.1. Summary of Changes from RFC 8321 🍓

This document defines the Alternate-Marking Method. It fixes ambiguities and builds on the experimental phase that was based on the original specification [RFC8321] 🧁

The relevant changes are:

  • Added recommendations about which methods to use when one or two flag bits are available for marking (Section 7) 💜
  • Restructured the doc to improve readability ✨
  • Removed the wording about the method's initial experiments, plus considerations that no longer apply 🙈
  • Extended the description of detailed aspects of the methodology, e.g., synchronization, timing, packet fragmentation, and marked and unmarked traffic handling 🔥

It's important to note that all the changes are totally backward compatible with [RFC8321], and no new additional technique has been introduced in this document compared to [RFC8321] 🫶

1.2. Requirements Language 📏

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🎀

2. Overview of the Method 🔭

To measure packet loss on a production traffic flow, there are different approaches. The most intuitive one is numbering the packets so that each router that receives the flow can immediately detect a missing packet 🕵️‍♀️ In theory that's super simple, but in practice it's not: it means inserting a sequence number into each packet, and the devices have to extract the number and check it in real time. That can be hard on live traffic 😩 If UDP is the transport protocol, there is no sequence number available. And if a higher-layer sequence number is used (e.g., in the RTP header), extracting it from each packet and processing it in real time could overload the device 💀

Another approach is to count the packets sent at one end, count the packets received at the other end, and compare the two values. This is much simpler to implement, but the devices doing the measurement have to be in sync: to compare two counters, they must refer to exactly the same set of packets 🧠 A flow is continuous and can't be stopped when a counter needs reading, so it can be hard to pin down exactly when to read it ⏰ One fix is to virtually split the flow into consecutive blocks by periodically inserting a delimiter, so each counter refers to exactly the same block of packets. The delimiter could be, for example, a special packet artificially inserted into the flow 🧚 But delimiting the flow with specific packets has limitations. First, it means generating additional packets within the flow, and the equipment has to be able to process them. Also, the method is vulnerable to out-of-order reception of delimiting packets and, to a lesser extent, to their loss 😬

The method in this document follows the second approach, but it doesn't use additional packets to virtually split the flow into blocks 🦋 Instead, it "marks" the packets, so packets in the same block share the same notional "color" while consecutive blocks get different colors 🌈 Each color change acts as an auto-synchronization signal that makes the measurements taken by different devices along the path more consistent 💖

Figure 1 shows a very simple network and how the method can measure packet loss on different network segments. By enabling measurement on several interfaces along the path, you can do link monitoring, node monitoring, or end-to-end monitoring 🎯 The method is flexible enough to measure packet loss on any segment of the network, so you can isolate the faulty element 🔍

                             Traffic Flow
      ========================================================>
        +------+       +------+       +------+       +------+
    ---<>  R1  <>-----<>  R2  <>-----<>  R3  <>-----<>  R4  <>---
        +------+       +------+       +------+       +------+
        .              .      .              .       .      .
        .              .      .              .       .      .
        .              <------>              <------->      .
        .          Node Packet Loss      Link Packet Loss   .
        .                                                   .
        <--------------------------------------------------->
                         End-to-End Packet Loss

                    Figure 1: Available Measurements

3. Detailed Description of the Method 🎀

This section describes, in detail, how the method operates 💅 Special emphasis goes to packet-loss measurement, since that's the core application of the method, but applicability to delay and jitter measurements is covered too ✨

More of the juicy details (3.1 onward) are coming in part 2 🫶

3.1. Packet-Loss Measurement ✨

okay so the big idea 💖 is to virtually chop a traffic flow into back-to-back blocks. Each block is a measurable chunk that every network device on the path can recognize without any confusion 🧚. If every device counts the packets in each block and we compare the numbers, we can tell whether packets went missing in any single block between any two points 👑.

As we covered last section, a simple way to make the blocks is to "color" the traffic (two colors are enough 💅). Packets in alternate consecutive blocks get different colors. Whenever the color flips, the old block ends and a new one begins. So everything in one block shares a color, and neighboring blocks never do. How many packets end up in each block depends on how you decide when to flip:

  • if the color flips after a fixed number of packets, every block holds the same number of packets (except for whatever got lost); and
  • if the color flips on a fixed timer, the number of packets in each block can differ depending on the packet rate.

Using a fixed timer to create the blocks is REQUIRED when you implement this spec 🫶. Switching after a fixed number of packets is an extra option, but its detailed spec is out of scope 🙈. There's an example of this in [EXPLICIT-FLOW-MEASUREMENTS].

Here's how a flow looks when it gets split into colored blocks 🌈:

   A: packet with A coloring
   B: packet with B coloring

            |           |           |           |           |
            |           |    Traffic Flow       |           |
    ------------------------------------------------------------------->
     BBBBBBB AAAAAAAAAAA BBBBBBBBBBB AAAAAAAAAAA BBBBBBBBBBB AAAAAAA
    ------------------------------------------------------------------->
       ...  |  Block 5  |  Block 4  |  Block 3  |  Block 2  |  Block 1
            |           |           |           |           |

Figure 2: Traffic Coloring

Figure 3 shows how this method can measure link packet loss between two adjacent nodes 🦋. Say we want to keep an eye on packet loss on the link between two routers, R1 and R2. The traffic gets colored alternately with two colors, A and B. Every time the color changes, you get a square-wave-looking signal, like this:

   Color A   ----------+           +-----------+           +----------
                       |           |           |           |
   Color B             +-----------+           +-----------+
              Block n        ...      Block 3     Block 2     Block 1
            <---------> <---------> <---------> <---------> <--------->

                                Traffic Flow
            ===========================================================>
   Color   ...AAAAAAAAAAA BBBBBBBBBBB AAAAAAAAAAA BBBBBBBBBBB AAAAAAA...
            ===========================================================>

Figure 3: Computation of Link Packet Loss

Coloring the traffic can be done by R1 itself if the traffic isn't already colored 💜. R1 needs two counters on its egress interface, C(A)R1 and C(B)R1. C(A)R1 counts the packets colored A, and C(B)R1 counts the ones colored B. While the traffic is colored A, only C(A)R1 goes up and C(B)R1 stays put. When it's colored B, only C(B)R1 goes up. C(A)R1 and C(B)R1 can then serve as reference values for working out packet loss from R1 to any other measurement point down the path ⭐. R2 needs two counters too, C(A)R2 and C(B)R2, on its ingress interface. These count the packets it receives on that interface that are colored A and B. When an A block ends, you can compare C(A)R1 with C(A)R2 and work out the packet loss in that block. Same deal with B: when the next B block ends, compare C(B)R1 with C(B)R2, and so on for every block after that 🎀.

You can do something similar with two counters on the R2 egress interface. Those count the packets sent out of R2's interface, and they become reference values for working out packet loss from R2 to any measurement point downstream of R2 🌸.

You can make the blocks as long as you like, which makes collecting and comparing the measurements from different devices easier 🍓. It's better not to read the counters right after the color switch, though. Some packets might arrive out of order and bump the counter of the previous block (color), so it's worth waiting a bit 🙈. A safe choice is to wait L/2 time units after the color switch before reading the counter of the previous color (where L is the duration of each block; see Section 5). The catch is that the longer each block lasts, the less often you can take a measurement 😭.

There are two strategies you can use when you implement the method 👑:

  • flow-based: the flow-based strategy is for when you need to monitor well-defined traffic flows. With this strategy, only the specified flow gets colored. Counters for packet-loss measurements can be set up for each individual flow or for the set as a whole, depending on how granular you want to be. This approach means you have to know in advance the path that the flows you're measuring take. Path rerouting and traffic load balancing need to be taken into account 💅.
  • link-based: measurements are done on all the traffic on a link-by-link basis. The link could be a physical link or a logical one. Counters could be set up for all the traffic as a whole, or for each traffic class (if you want to monitor each class separately). But if you go with the per-class option, you need two counters for each class ✨.

The flow-based strategy is REQUIRED when you implement this spec 🫶. It requires identifying the flow you want to monitor and discovering the path that flow takes. You can monitor a single flow or several flows grouped together. With a group, though, measurements only stay consistent if all the flows in the group follow the same path 🦋. Also, if you measure a group of flows together, you can't work out exactly which flow was hit by packet loss 🙈. To get measurements per individual flow, you need to set up counters for each specific flow 💖. Once you've identified the flow(s) to monitor, you have to configure the monitoring on the right nodes. Configuring the monitoring means setting up a rule to catch the traffic and setting up counters to count the packets 🧚. For just end-to-end monitoring, it's enough to turn monitoring on at the first- and last-hop routers of the path. The mechanism is completely transparent to intermediate nodes and doesn't depend on the path the traffic takes 🌈. But if you want hop-by-hop monitoring along the entire path, you have to turn monitoring on at every node from source to destination 🔥. If the exact path a flow takes isn't known ahead of time (say the flow has multiple paths to reach its destination), you have to turn monitoring on along every path. Counters on the interfaces the flow goes through will report a packet count, while counters on the other interfaces will be null 💀.

3.2. One-Way Delay Measurement 🦋

The same principle we used for packet loss also works for one-way delay measurement 💜. There are two methodologies, described below.

Heads up: for all the one-way delay alternatives in the next sections, you can add up the one-way delays of the two directions of a path to get the two-way delay (the round-trip "virtual" delay) ✨. For the timestamp formats, you can use the Network Time Protocol (NTP) [RFC5905] or the IEEE 1588 Precision Time Protocol (PTP) [IEEE-1588] (both discussed in the previous section), depending on how much precision you need 🕰️.

3.2.1. Single-Marking Methodology 💅

The alternation of colors can act as a time reference for calculating delay 🎀. Whenever the color changes (meaning a new block has started), a network device can store the timestamp of the first packet of that new block. You compare that timestamp with the timestamp of the same packet on a second router to work out the packet delay 🫶. Looking at Figure 2, R1 stores the timestamp TS(A1)R1 when it sends the first packet of block 1 (A-colored). It stores TS(B2)R1 when it sends the first packet of block 2 (B-colored), and so on for every other block. R2 does the same on the receiving side, recording TS(A1)R2, TS(B2)R2, and so on 🌸. Because each timestamp refers to a specific packet (the first packet of each block), if there's no packet loss or misordering, we can be sure the timestamps being compared to calculate delay refer to the same packets 👑. By comparing TS(A1)R1 with TS(A1)R2 (and likewise TS(B2)R1 with TS(B2)R2, and so on), you can measure the delay between R1 and R2 🧚. To get more measurements, you can take and store more timestamps that refer to other packets within each block. You could take a timestamp every N packets, which gives you multiple delay measurements 🍓. Taking this to the extreme, in principle you could measure the delay of every single packet by taking and comparing the matching timestamps. That's possible, but totally impractical from an implementation point of view 😭.

To compare timestamps collected on different routers in a coherent way, the clocks on the network nodes MUST be in sync (Section 5) ⏰. Also, a measurement is only valid if no packet loss occurs and packet misordering can be avoided. Otherwise, the first packet of a block on R1 might not be the first packet of the same block on R2. For example, that packet could get lost between R1 and R2, or it could arrive after the next packet 🙈. Packet misordering is generally undetectable, so there's no way to check whether the first packet on R1 is the same one on R2. That uncertainty is part of the intrinsic error in this measurement 💖.

3.2.1.1. Mean Delay

okay so the old delay method is super sensitive to packets arriving out of order 😭 so bestie, a mean delay approach can be considered 💅 the idea is you take the average arrival time of the packets in a single block. each device locally stores a timestamp for every packet it gets in that block, adds them all up, and divides by how many packets arrived ✨ that gives you the average arrival time for the block. then you subtract the average arrival times of two neighboring devices and boom, you've got the mean delay between those nodes 🦋

this is lowkey iconic because it cuts down how many timestamps you have to collect (just one per block for each device!), and it's robust to out-of-order packets with only a small error when packets get lost 🫶 but ngl, the measurement error can get augmented by piling up the error from a lot of packets 🙈 also, it only gives you one number for the whole block, and it doesn't tell you the minimum, maximum, or median delay values [RFC6703] 💀 you could fix that by making the blocks shorter (like from minutes down to seconds), but that means a super optimized implementation. so the mean delay calculation may not be so viable in some cases 🥺

3.2.2. Double-Marking Methodology

okay so we already said the Single-Marking methodology for one-way delay has limits: it's sensitive to out-of-order packets, and even the mean delay calculation doesn't tell you anything about how the delay values are distributed across the block 😩 it would be useful to have the minimum, maximum, and median delay values too, and honestly to know more about the statistical distribution of delay values in general 📊 so to get more info and dodge the out-of-order drama, we use a Double-Marking methodology 👑

the vibe is: use the first marking to create the alternate flow, and then, inside that colored flow, use a second marking to pick out the packets you measure delay/jitter on 🎀 the first marking is needed for packet loss and may be used for mean delay measurement. the second marking creates a new set of marked packets that are fully identified over the network, so a network device can store their timestamps 🧚 you compare those timestamps with the timestamps of the same packets on the next node to compute delay values for each packet. you can increase the number of measurements by changing how often the second marking happens, but the frequency of the second marking must not be too high or you'll run into out-of-order issues 🙅‍♀️ there should be an adequate time gap between packets with the second marking, to avoid out-of-order issues and to have a number of measurement packets that are rate independent 🌸 this gap may be, at the minimum, the mean network delay calculated with the previous methodology. so you can pick a proper time gap that guarantees a fixed number of double-marked packets, uniformly spaced in each block ✨

if second-marked packets get lost, it's easy to recognize since the number of double-marked packets is known for each block 👀 based on the spacing between them, it can also be possible to figure out which packet of the second marking sequence was lost, and then perform the measurements only for the remaining packets. but this may get complicated if more packets are lost 😵 in that case, an implementation may simply discard the delay measurements for the corrupted block and proceed with the next block 🫶

An efficient and robust mode is to select a single packet with the second marking for each block. that way, there's no time gap to consider between double-marked packets to avoid their reorder 💅 it's also easier to identify the only double-marked packet in each block and skip the delay measurement for that block if it's lost 🦋

The Double-Marking methodology can also be used to get more statistics on delay extent data, e.g., percentiles, variance, and median delay values 🌈 a subset of batch packets is selected for extensive delay calculation by using the second marking, so it's possible to do a detailed analysis on these double-marked packets. there are classic algorithms for median and variance calculation, but they're out of the scope of this document 🙈 the conventional range (maximum-minimum) should be avoided for several reasons, including the stability of the maximum delay, which is influenced by outliers. Section 6.5 of [RFC5481] highlights how the 99.9th percentile of delay and delay variation is more helpful to performance planners 💜

3.3. Delay Variation Measurement

Similar to one-way delay measurement (both for Single Marking and Double Marking), the method can also be used to measure the inter-arrival jitter 🎀 we refer to the definition in [RFC3393]. the alternation of colors, for a Single-Marking method, can be used as a time reference to measure delay variations 🌈 in the case of Double Marking, the time reference is given by the second-marked packets. considering the example depicted in Figure 2, R1 stores the timestamp TS(A)R1 whenever it sends the first packet of a block, and R2 stores the timestamp TS(B)R2 whenever it receives the first packet of a block 🧚 the inter-arrival jitter can be easily derived from one-way delay measurement, by evaluating the delay variation of consecutive samples ✨

the concept of mean delay can also be applied to delay variation, by evaluating the average variation of the interval between consecutive packets of the flow from R1 to R2 💖

4. Alternate-Marking Functions 🌸

4.1. Marking the Packets

the coloring operation is fundamental in order to create packet blocks and marked packets 🎨 this means choosing where to activate the coloring and how to color the packets.

in the case of flow-based measurements, the flow to monitor can be defined by a set of selection rules (e.g., header fields) used to match a subset of the packets. this way, it's possible to control the number of nodes involved, the path followed by the packets, and the size of the flows 🧠 it's possible, in general, to have multiple coloring nodes, or a single coloring node that is easier to manage and doesn't raise any risk of conflict. coloring in multiple nodes can be done, and the requirement is that the coloring must change periodically between the nodes according to the timing considerations in Section 5, so every node that is designated as a measurement point along the path should be able to identify the colored packets unambiguously 🫶 furthermore, [RFC9342] generalizes the coloring for multipoint-to-multipoint flow. in addition, it can be advantageous to color the flow as close as possible to the source, because it allows an end-to-end measure if a measurement point is enabled on the last-hop router as well 🦋

for link-based measurements, all traffic needs to be colored when transmitted on the link. if the traffic had already been colored, then it has to be re-colored, because the color must be consistent on the link 🔗 this means that each hop along the path must (re-)color the traffic; the color is not required to be consistent along different links.

traffic coloring can be implemented by setting specific flags in the packet header and changing the value of that bit periodically 🌈 how to choose the marking field depends on the application and is out of scope here 💅

4.2. Counting and Timestamping Packets

for flow-based measurements, assuming that the coloring of the packets is performed only by the source nodes, the nodes between source and destination (inclusive) have to count and timestamp the colored packets that they receive and forward ⏱️ this operation can be enabled on every router along the path or only on a subset, depending on which network segment is being monitored (a single link, a particular metro area, the backbone, or the whole path). since the color switches periodically between two values, two counters (one for each value) are needed for each flow and for every interface being monitored 🔢 the number of timestamps to be stored depends on the method for delay measurement that is applied. furthermore, [RFC9342] generalizes the counting for multipoint-to-multipoint flow 💜

in the case of link-based measurements, the behavior is similar, except that the coloring, counting, and timestamping operations are performed on a link-by-link basis at each endpoint of the link 🔗

another important consideration is when to read the counters or when to select the packets to be double-marked for delay measurement 🕰️ it involves timing aspects that are further described in Section 5 ✨

4.3. Data Collection and Correlation 📬

okay so the nodes that are allowed to do performance monitoring collect the counter and timestamp values, but they can't use that info alone to measure packet loss and delay, bestie. they only have their own samples 🥺

data collection means the counters and timestamps get sent out as soon as they've been read. data correlation is the mechanism that compares those counters and timestamps so you can work out packet loss, delay, and delay variation 💅✨

there are two main ways to do both collection and correlation, and which one you pick depends on the Alternate-Marking application and use case 🎀

  • Centralized with an NMS. The Network Management System (NMS) does the correlating. This can run in Push Mode or Polling Mode. In Push Mode, each router periodically sends its info to the NMS. In Polling Mode, the NMS periodically polls the routers to collect it 🫶
  • Protocol-based distributed. The endpoints swap counter and timestamp values with each other. You can do this by making a brand-new protocol or by extending an existing one, like the Two-Way Active Measurement Protocol (TWAMP) defined in [RFC5357] or the One-Way Active Measurement Protocol (OWAMP) defined in [RFC4656], so nodes can share the counters and timestamps 🦋

the next paragraphs walk through an example data correlation mechanism. it can be used no matter which solution you go with 🌸

when data gets collected on the upstream and downstream nodes (like packet counts for packet-loss measurement, or timestamps for packet delay measurement) and then reported to or pulled by other nodes or an NMS, a certain data correlation mechanism SHOULD be in use. its job is to help the nodes or NMS tell whether two or more packet counts belong to the same block of markers, or whether two timestamps belong to the same marked packet 🔍💜

the Alternate-Marking Method splits the packets of the measured flow into different measurement blocks, quite literally 🍓 an implementation MAY use a Block Number (BN) for data correlation. if it does, the BN MUST be assigned to each measurement block and associated with each packet count and timestamp that gets reported to or pulled by other nodes or NMSs. say the nodes or NMS see the same BNs attached to a packet count from an upstream node and another from a downstream node. then they treat those two packet counts as the same block 🧚

the BN mechanism assumes the measurement nodes are time synchronized, which means they need some kind of time sync capability, like NTP [RFC5905] or IEEE 1588 PTP [IEEE-1588] 🕰️✨

5. Synchronization and Timing ⏰

color switching is the reference for every network device acting as a measurement point. the only thing that has to be achieved is that they recognize the right batch along the path, so they can grab the related counters and timestamps 🌈

real talk, clocks in network devices aren't accurate. that means there's a clock error between measurement points R1 and R2 😭 to make the methodology work, they MUST be synchronized to the same clock reference with adequate accuracy, so every device matches the marking bit to the correct block consistently. and that's not all. packet reordering is also super common in packet networks because of equal-cost multipath (ECMP) 🙈 the delay between measurement points is the main cause of out-of-order packets, since each packet can be delayed differently. if the block is big enough, reordering only happens at the edges of adjacent blocks, and then it's easy to assign the reordered packets to the right interval blocks 💅

so in summary, two things matter: the clock error between network devices, and how long we have to wait so packets aren't out of order because of network delay ⭐

here's the figure that shows both issues 👑

...BBBBBBBBB | AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA | BBBBBBBBB...
             |<======================================>|
             |                   L                    |
...=========>|<==================><==================>|<==========...
             |       L/2                   L/2        |
             |<===>|                            |<===>|
                d  |                            |   d
                   |<==========================>|
                    available counting interval

                       Figure 4: Timing Aspects

where L is the time duration of each block 🦋

all network devices are assumed to be synchronized to a common reference time with an accuracy of +/- A/2. so the difference between the clock values of any two network devices is bounded by A 🫶

the network delay between network devices can be modeled as a normal distribution, where 99.7% of the samples fall within 3 standard deviations of the average 📊

the guard band d is calculated like this 🧮

d = A + D_avg + 3*D_stddev

where A is the clock accuracy, D_avg is the average network delay between the network devices, and D_stddev is the standard deviation of the delay ✨

the available counting interval is L - 2d, and that has to be > 0, no cap 🙌

now the big one. the condition that MUST be satisfied, and it's a requirement on synchronization accuracy, is:

d < L/2

this is the fundamental rule for deciding when to read the counters and when to choose the packets to be double-marked 🔥 basically, packet counters and double-marked packets MUST be taken and chosen, respectively, within the available counting interval that isn't affected by error factors. no sneaking in outside it, periodt 💀

if the time duration L of each block isn't super small, the synchronization requirement can be met even with a relatively inaccurate sync method 🌸

6. Packet Fragmentation 🧩

fragmentation can be handled with the Alternate-Marking Method using the guidance below 💖

  • Marking nodes MUST mark all fragments if there are flag bits to use (meaning it's in the specific encapsulation), and they should be treated as if they were separate packets 🎀
  • Nodes that fragment packets within the measurement domain SHOULD, if they have the capability to do so, make sure only one resulting fragment carries the marking bit(s) of the original packet. skipping this can introduce errors into the measurement 😬
  • Measurement points SHOULD simply ignore unmarked fragments and count marked fragments as full packets. but if resources allow, measurement points MAY take note of both marked and unmarked initial fragments, and only increment the corresponding counter if (a) other fragments are also marked or (b) it observes all other fragments and they're all unmarked 🧠

the proposed approach lets the marking node mark all the fragments, except when fragmentation happens within the network domain. in that case, it's suggested to mark only the first fragment 🦋

7. Recommendations for Deployment 🚀

the methodology from the previous sections works for all kinds of performance measurement problems. the only requirement is to select and mark the flow you want to monitor. that way, the sender batches packets, and each batch gets alternately marked so the receiver can spot it easily 💅 [RFC8321] has experimental examples, and [IEEE-NETWORK-PNPM] also shares some info about deployment experience 📚

different deployments might have either one or two flag bits available for marking 🌈

One flag: packet-loss measurement MUST be done as described in Section 3.1, and delay measurement MUST be done according to the Single-Marking Method described in Section 3.2.1. mean delay (Section 3.2.1.1) MAY also be used, but it could mean more computational load 🧮

Two flags: packet-loss measurement MUST be done as described in Section 3.1, and delay measurement MUST be done according to the Double-Marking Method described in Section 3.2.2. in this case, Single Marking MAY also be used alongside Double Marking. the two approaches give slightly different pieces of info, which can be combined for a more robust data set 👑

there are some operational guidelines to think about when you're deciding whether to follow the recommendations above and use one or two flags 💜

  • The Alternate-Marking Method uses specific flags in the packet header, so the number of flags available for the implementation is a big deal 🫶 if there's only one flag, there's no other way. if there are two, the two-flag option is definitely more complete ✨
  • The duration of the Alternate-Marking period affects how often you measure, and you can decide it based on the temporal sampling you need. but it can't be chosen freely, as explained in Section 5 ⏰
  • The Alternate-Marking methodologies let you calculate packet loss, delay, and delay variation, but depending on the method (Single Marking or Double Marking), you get different kinds of info. for example, if you want more statistics of extent data, the two-flag option is the one you want 📊 so the type of data you need in your scenario is another thing to factor in 🎀
  • The Alternate-Marking Methods come with different computational loads depending on the method, so the computing resources on the measurement points can also shape your choice. mean delay calculation, for example, may need more processing, so it might not be the best pick if you're trying to keep the load down 💻

the experiment with Alternate-Marking methodologies confirmed the benefits already described in [RFC8321] 🥹

a deployment of the Alternate-Marking Method should also think about how to handle and recognize marked and unmarked traffic. since Alternate Marking normally uses a marking field that's dedicated, reserved, and included in a protocol extension, the measurement points can tell whether measurement is turned on by checking if that specific extension is in the packets or not 🔍

it's worth mentioning some related work too. [IEEE-NETWORK-PNPM] explains the Alternate-Marking Method alongside new mechanisms based on hashing techniques 🌸

7.1. Controlled Domain Requirement 🎀

okay so the Alternate-Marking Method is the perfect example of a solution that only works in a controlled domain [RFC8799] 💅

A controlled domain is basically a managed network that picks who gets in, watches what's happening, and locks down access by enforcing policies at its edges. 🚪 The boundary guards throw out unwanted outside packets coming in, and they check internal packets heading out. It doesn't have to be one admin domain or one organization, ngl. A controlled domain can be a single administrative domain, or several of them working under one defined network management setup 🤝 You do need to be able to control those boundaries, though, and you need extra protection like authentication, encryption, and integrity protection if your traffic ever crosses the open Internet 🔐

For security reasons, the Alternate-Marking Method MUST only be applied to controlled domains. 🚨 No exceptions, bestie!

8. Compliance with Guidelines from RFC 6390 💜

[RFC6390] is the framework and process for writing Performance Metrics for protocols above and below the IP layer 🧚 (think IP-based apps running over reliable or datagram transports).

This document isn't trying to invent a new Performance Metric 🙅‍♀️ It's a new Method of Measurement for a few Performance Metrics that are already standardized. Still, it's a good idea to apply the [RFC6390] guidelines here so the method gets a more complete and coherent description ✨ The mechanisms in this doc use the Performance Metric Definition template from Section 5.4 of [RFC6390], together with the Dependencies laid out in Section 5.5 of that same document.

  • Metric Name / Metric Description: as I already said, no new Performance Metrics here 💅 Instead, this describes a fresh way to measure packet loss [RFC7680]. The same concept, with small tweaks, can also measure delay [RFC7679] and jitter [RFC3393]. The document mainly focuses on applying it to packet-loss measurement 🌸
  • Method of Measurement or Calculation: following the method from the earlier sections, you work out the number of lost packets by subtracting the counter value on the source node from the counter value on the destination node 🧮 Both counters have to refer to the same color. The calculation happens once the counter values are in a steady state. That steady state is built into the marking method's counters, because switching colors keeps the counter for each color inactive for the length of a marking period 🦋
  • Units of Measurement: the method calculates and reports the exact number of packets that the source node sent but the destination node never received 🎯
  • Measurement Point(s) with Potential Measurement Domain: you can measure between adjacent nodes (per link), or along a multi-hop path, as long as the traffic being measured actually follows that path 🛣️ On a multi-hop path, you can measure end to end and hop by hop too 💖
  • Measurement Timing: the method has a constraint on how often you can measure 😬 Section 5 covers this in detail. The marking period and the guard band interval are tightly linked so you avoid out-of-order issues 🙈 That's because a measurement needs the counter to be in a steady state, and that only happens while the traffic is being colored with the alternate color 🌈
  • Implementation: the method uses one or two marking bits to color packets 🎀 That means you can use policy configurations on the router to color packets and set up the counter for each color to match. The path the measured traffic takes should be known in advance, so you can configure the counters along it and compare the right values 🧭
  • Verification: the methodology has been tested and deployed experimentally, in both lab and operational network scenarios 🧪 The tests measured packet loss and delay on traffic patterns made by traffic generators, plus precision test instruments and network emulators 🔬
  • Use and Applications: the method can measure packet loss with high precision on live traffic 🔥 Plus, by combining end-to-end and per-link measurements, it can pinpoint the exact link that's experiencing loss events 🕵️‍♀️
  • Reporting Model: the counter values need to be sent to a centralized management system that does the calculations 🖥️ Each sample has to include a reference to the time interval it belongs to, so the management system can correlate things correctly 🕰️ Samples should be sent while the corresponding counter is in a steady state (within a time interval). Otherwise, the sample value should be stored locally 📦
  • Dependencies: the counter values have to be correlated with the time interval they refer to ⏱️
  • Organization of Results: the Method of Measurement produces singletons, as defined in [RFC2330] 💜
  • Parameters: the main parameters are the info about the flow or link being measured, the time interval used to alternate the colors and read the counters, and the type of method chosen for packet-loss and delay measurements 📋

9. IANA Considerations 🌷

This document has no IANA actions. Nothing to register, bestie 💅

10. Security Considerations 🔐

This document describes a method for measurements that doesn't directly affect Internet security or the applications running on the Internet 🙌 But implementing it still means being really mindful of security and privacy concerns 😤

There are two kinds of security concerns: potential harm caused by the measurements, and potential harm to the measurements. 💀

  • Harm caused by the measurement: the measurements in this document are Passive, so no new packets get injected into the network. That means no new potential harm to the network itself or to data traffic 🙏 Still, the method does change a header or encapsulation of data packets on the fly. That has to be done without degrading the quality of service for the packets being measured, and it has to keep routers doing the measuring stable and performing well ⚡ Network reconnaissance is one of the big threats in Operations, Administration, and Maintenance (OAM) protocols. An attacker can learn about network performance just by passively eavesdropping on OAM messages 👀 The good news is that the marking bits are the only information exchanged between network devices in these methods. So passively eavesdropping on data plane traffic doesn't let an attacker learn anything about network performance 🦋
  • Harm to the Measurement: the measurements could be messed up by routers altering packet markings, or by an attacker injecting fake traffic 😭 Authentication techniques, like digital signatures, may be used where appropriate to guard against injected traffic attacks 🔑 Routers and other network devices along the path can intentionally change the marking bits, as mentioned above. So this mechanism should only be applied within a controlled domain. That way the routers and other devices are locally administered, and this kind of attack can be avoided 🛡️

An attacker from outside the controlled domain can maliciously send marked packets 👿 But no problems happen if Alternate Marking isn't supported in the controlled domain 🙅‍♀️ If it is supported there, the measurements have to be protected from being affected. So externally marked packets must be checked for markings, and then filtered or cleared if needed 🧹

The precondition for applying the Alternate-Marking Method is that it MUST be applied in specific controlled domains. That confines potential attack vectors to within the network domain 🏰 A limited administrative domain gives the network administrator the means to select, monitor, and control access to the network, which makes it a trusted domain 🤍 The domain is expected to enforce policies at its boundaries, filtering both external marked packets coming in and internal marked packets going out 🚧 So a trusted domain is unlikely to suffer packet hijacking, because marked packets are only processed and used inside the controlled domain 🙌 Still, leakages can happen for various reasons, like a failure or a fault 😬 In that case, nodes outside the domain are expected to ignore marked packets, since they aren't configured to handle them, and they should not process them 🚫

It might be theoretically possible to modulate the marking to act as a covert channel for an on-path observer 🕵️‍♀️ That could affect both the data and management planes. But again, applying this to a controlled domain helps reduce the effects 🌈

It's worth highlighting that an attacker can't learn about network performance from a single monitoring point ✨ They'd need synchronized monitoring points at multiple places along the path, because they'd have to do the same kind of measurement and aggregation that Service Providers using Alternate Marking have to do 🧠

Attacks on the data collection and reporting of statistics between the monitoring points and the NMS can break the system's proper functioning 💔 So the channels used to report back flow statistics MUST be secured 🔒

The privacy concerns of network measurement are pretty limited, because the method only relies on information in the header or encapsulation and doesn't release any user data 🫶 Header or encapsulation info is metadata that could compromise user privacy in principle. But the limited marking technique in this document seems unlikely to substantially increase existing privacy risks from that metadata 🙂 It might be theoretically possible to modulate the marking as a covert channel, but its data rate would be very low if it's going to avoid adversely affecting the measurement systems that watch the marking 🐢

Delay attacks are another potential threat here 😱 Delay measurement uses one specific packet in each block, marked by a dedicated color bit. So an on-path attacker can selectively add synthetic delay only to delay-colored packets, which causes systematic error in the delay measurements 🫠 As discussed in earlier sections, the methods in this document rely on an underlying time synchronization protocol. So by attacking that time protocol, an attacker could potentially compromise the integrity of the measurement ⏰ A detailed discussion of threats against time protocols, and how to mitigate them, is in [RFC7384] 📚

11. References 📖

11.1. Normative References 📚

okay so these are the docs this spec leans on, bestie 🫶

11.2. Informative References 🌸

these are the extra reads for when you want the full story, no pressure 🧚

Acknowledgements 🙌

The authors would like to thank Alberto Tempia Bonda, Luca Castaldelli, and Lianshu Zheng for their contribution to the experimentation of the method.

The authors would also like to thank Martin Duke and Tommy Pauly for their assistance and their detailed and precious reviews. 💜

Contributors 💖

  • Xiao Min
    • ZTE Corp.
    • Email: xiao.min2@zte.com.cn
  • Mach(Guoyi) Chen
    • Huawei Technologies
    • Email: mach.chen@huawei.com
  • Alessandro Capello
    • Telecom Italia
    • Email: alessandro.capello@telecomitalia.it

Authors' Addresses 🦋

Giuseppe Fioccola (editor)
Huawei Technologies
Riesstrasse, 25
80992 Munich
Germany
Email: giuseppe.fioccola@huawei.com
Mauro Cociglio
Telecom Italia
Email: mauro.cociglio@outlook.com
Greg Mirsky
Ericsson
Email: gregimirsky@gmail.com
Tal Mizrahi
Huawei Technologies
Email: tal.mizrahi.phd@gmail.com
Tianran Zhou
Huawei Technologies
156 Beiqing Rd.
Beijing
100095
China
Email: zhoutianran@huawei.com
📦 where this page comes from