RFC 7325

MPLS Forwarding Compliance and Performance Requirements

📅 published August 2014, from ietf

MPLS Forwarding Compliance and Performance Requirements 🎀

okay so this RFC is literally the MPLS forwarding checklist that implementers need, and it's giving "read this before you ship" energy 💅✨

Quick facts from the header: 👑

Number: RFC 7325
Series: IETF Request for Comments
Authors: C. Villamizar (Ed.) of OCCNC, K. Kompella of Juniper Networks, S. Amante of Apple Inc., A. Malis of Huawei, and C. Pignataro of Cisco
Category: Informational, so it's published for info and is not a Standards Track spec 🧚
ISSN: 2070-1721
Date: August 2014
Obsoletes/Updates: the header doesn't list any RFCs it obsoletes or updates 🌸

Abstract

okay so the whole point of this doc is to give implementers guidelines on MPLS forwarding, and also to give everyone a basis for evaluating forwarding implementations 🦋 It covers a lot of different MPLS forwarding stuff. The big thing it does is call out the practical requirements people tend to miss, either because they're never actually stated or they're barely emphasized. Some of these are technically optional for conformance but providers still treat them as mandatory, so yeah, they're kind of a big deal 🔥

Status of This Memo

ok so this one is NOT a standards track spec, it's just here for information 📚

It is a product of the IETF 🌈 and it reflects what the IETF community agrees on. It got public review and was approved for publication by the IESG. Not every IESG-approved doc can become any level of Internet Standard, so check Section 2 of RFC 5741 if you're curious about that 💜

For the current status, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc7325 🍓

Copyright Notice

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can find at http://trustee.ietf.org/license-info (the version in effect on the publication date). Please actually read those, bestie, because they spell out your rights and restrictions for this document 🙈

If you pull Code Components out of this doc, they have to include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They also come with no warranty, per the Simplified BSD License 💖

   1.  Introduction and Document Scope . . . . . . . . . . . . . . .   4
     1.1.  Abbreviations . . . . . . . . . . . . . . . . . . . . . .   4
     1.2.  Use of Requirements Language  . . . . . . . . . . . . . .   8
     1.3.  Apparent Misconceptions . . . . . . . . . . . . . . . . .   9
     1.4.  Target Audience . . . . . . . . . . . . . . . . . . . . .  10
   2.  Forwarding Issues . . . . . . . . . . . . . . . . . . . . . .  11
     2.1.  Forwarding Basics . . . . . . . . . . . . . . . . . . . .  11
       2.1.1.  MPLS Special-Purpose Labels . . . . . . . . . . . . .  12
       2.1.2.  MPLS Differentiated Services  . . . . . . . . . . . .  13
       2.1.3.  Time Synchronization  . . . . . . . . . . . . . . . .  14
       2.1.4.  Uses of Multiple Label Stack Entries  . . . . . . . .  14
       2.1.5.  MPLS Link Bundling  . . . . . . . . . . . . . . . . .  15
       2.1.6.  MPLS Hierarchy  . . . . . . . . . . . . . . . . . . .  16
       2.1.7.  MPLS Fast Reroute (FRR) . . . . . . . . . . . . . . .  16
       2.1.8.  Pseudowire Encapsulation  . . . . . . . . . . . . . .  17
         2.1.8.1.  Pseudowire Sequence Number  . . . . . . . . . . .  17
       2.1.9.  Layer 2 and Layer 3 VPN . . . . . . . . . . . . . . .  19
     2.2.  MPLS Multicast  . . . . . . . . . . . . . . . . . . . . .  20
     2.3.  Packet Rates  . . . . . . . . . . . . . . . . . . . . . .  21
     2.4.  MPLS Multipath Techniques . . . . . . . . . . . . . . . .  23
       2.4.1.  Pseudowire Control Word . . . . . . . . . . . . . . .  24
       2.4.2.  Large Microflows  . . . . . . . . . . . . . . . . . .  24
       2.4.3.  Pseudowire Flow Label . . . . . . . . . . . . . . . .  25
       2.4.4.  MPLS Entropy Label  . . . . . . . . . . . . . . . . .  25
       2.4.5.  Fields Used for Multipath Load Balance  . . . . . . .  25
         2.4.5.1.  MPLS Fields in Multipath  . . . . . . . . . . . .  26
         2.4.5.2.  IP Fields in Multipath  . . . . . . . . . . . . .  27
         2.4.5.3.  Fields Used in Flow Label . . . . . . . . . . . .  29
         2.4.5.4.  Fields Used in Entropy Label  . . . . . . . . . .  29
     2.5.  MPLS-TP and UHP . . . . . . . . . . . . . . . . . . . . .  30
     2.6.  Local Delivery of Packets . . . . . . . . . . . . . . . .  30
       2.6.1.  DoS Protection  . . . . . . . . . . . . . . . . . . .  31
       2.6.2.  MPLS OAM  . . . . . . . . . . . . . . . . . . . . . .  33
       2.6.3.  Pseudowire OAM  . . . . . . . . . . . . . . . . . . .  34
       2.6.4.  MPLS-TP OAM . . . . . . . . . . . . . . . . . . . . .  34
       2.6.5.  MPLS OAM and Layer 2 OAM Interworking . . . . . . . .  35
       2.6.6.  Extent of OAM Support by Hardware . . . . . . . . . .  36
       2.6.7.  Support for IPFIX in Hardware . . . . . . . . . . . .  37
     2.7.  Number and Size of Flows  . . . . . . . . . . . . . . . .  37
   3.  Questions for Suppliers . . . . . . . . . . . . . . . . . . .  38
     3.1.  Basic Compliance  . . . . . . . . . . . . . . . . . . . .  38
     3.2.  Basic Performance . . . . . . . . . . . . . . . . . . . .  40
     3.3.  Multipath Capabilities and Performance  . . . . . . . . .  41
     3.4.  Pseudowire Capabilities and Performance . . . . . . . . .  41
     3.5.  Entropy Label Support and Performance . . . . . . . . . .  42
     3.6.  DoS Protection  . . . . . . . . . . . . . . . . . . . . .  42
     3.7.  OAM Capabilities and Performance  . . . . . . . . . . . .  42
   4.  Forwarding Compliance and Performance Testing . . . . . . . .  43
     4.1.  Basic Compliance  . . . . . . . . . . . . . . . . . . . .  43
     4.2.  Basic Performance . . . . . . . . . . . . . . . . . . . .  44
     4.3.  Multipath Capabilities and Performance  . . . . . . . . .  45
     4.4.  Pseudowire Capabilities and Performance . . . . . . . . .  46
     4.5.  Entropy Label Support and Performance . . . . . . . . . .  46
     4.6.  DoS Protection  . . . . . . . . . . . . . . . . . . . . .  47
     4.7.  OAM Capabilities and Performance  . . . . . . . . . . . .  47
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .  48
   6.  Organization of References Section  . . . . . . . . . . . . .  50
   7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  50
     7.1.  Normative References  . . . . . . . . . . . . . . . . . .  50
     7.2.  Informative References  . . . . . . . . . . . . . . . . .  53
   Appendix A.  Acknowledgements . . . . . . . . . . . . . . . . . .  59

1. Introduction and Document Scope ✨

so this doc started out as a response to concerns raised on the MPLS WG mailing list about shortcomings in MPLS forwarding implementations 📬 Writing down the existing misconceptions and pitfalls might help everyone stop repeating the same past mistakes 🙈 It's since grown into a much broader set of forwarding requirements, which is super iconic of it 🌸

The focus is MPLS forwarding, base pseudowire forwarding, and MPLS Operations, Administration, and Maintenance (OAM) 🦋 It talks about how to use the pseudowire Control Word and the pseudowire Sequence Number. Two things are out of scope: the specifics of the pseudowire Attachment Circuit (AC) and Native Service Processing (NSP), and particular pseudowire applications like the various forms of Virtual Private Network (VPN) 💅

MPLS support for multipath techniques is considered essential by lots of service providers, and it's useful for other high-capacity networks too 🔥 To get enough entropy out of MPLS, providers and others find it essential that the MPLS implementation interprets the MPLS payload as IPv4 or IPv6, based on the contents of the first nibble of the payload 🧚 Using IP addresses, the IP protocol field, and the UDP and TCP port number fields in multipath load balancing is in scope ✨ Other IP protocol fields, like tunneling protocols carried within IP, are out of scope 🫶

Implementation details are a local matter, so they're out of scope 🌈 Most interfaces today run at 1 Gb/s or faster. The doc assumes all forwarding operations happen in specialized forwarding hardware, not on a general-purpose processor. This is often called "fast path" and "slow path" processing 💜 Some recommendations are made about putting control or management-plane functionality in specialized hardware, or doing it with limited help from that hardware. The advice is based on the expected control and management protocol loads, and on the need for denial of service (DoS) protection 🍓

1.1. Abbreviations ✨

okay bestie, here's the whole alphabet soup 🍓 every acronym the doc uses, copied exactly so nothing gets lost

AC    Attachment Circuit ([RFC3985])

ACH   Associated Channel Header (pseudowires)

ACK   Acknowledgement (TCP flag and type of TCP packet)
AIS   Alarm Indication Signal (MPLS-TP OAM)

ATM   Asynchronous Transfer Mode (legacy switched circuits)

BFD   Bidirectional Forwarding Detection

BGP   Border Gateway Protocol

CC-CV Continuity Check and Connectivity Verification

CE    Customer Edge ([RFC4364])

CPU   Central Processing Unit (computer or microprocessor)

CT    Class Type ([RFC4124])

CW    Control Word ([RFC4385])

DCCP  Datagram Congestion Control Protocol

DDoS  Distributed Denial of Service

DM    Delay Measurement (MPLS-TP OAM)

DSCP  Differentiated Services Code Point ([RFC2474])

DWDM  Dense Wave Division Multiplexing

DoS   Denial of Service

E-LSP Explicitly TC-encoded-PSC LSP ([RFC5462])

EBGP  External BGP

ECMP  Equal-Cost Multipath

ECN   Explicit Congestion Notification ([RFC3168] and [RFC5129])

EL    Entropy Label ([RFC6790])

ELI   Entropy Label Indicator ([RFC6790])

EXP   Experimental (field in MPLS renamed to "TC" in [RFC5462])

FEC   Forwarding Equivalence Classes ([RFC3031]); also Forward Error
      Correction in other context

FR    Frame Relay (legacy switched circuits)
FRR   Fast Reroute ([RFC4090])

G-ACh Generic Associated Channel ([RFC5586])

GAL   Generic Associated Channel Label ([RFC5586])

GFP   Generic Framing Procedure (used in OTN)

GMPLS Generalized MPLS ([RFC3471])

GTSM  Generalized TTL Security Mechanism ([RFC5082])

Gb/s  Gigabits per second (billion bits per second)

IANA  Internet Assigned Numbers Authority

ILM   Incoming Label Map ([RFC3031])

IP    Internet Protocol

IPVPN Internet Protocol VPN

IPv4  Internet Protocol version 4

IPv6  Internet Protocol version 6

L-LSP Label-Only-Inferred-PSC LSP ([RFC3270])

L2VPN Layer 2 VPN

LDP   Label Distribution Protocol ([RFC5036])

LER   Label Edge Router ([RFC3031])

LM    Loss Measurement (MPLS-TP OAM)

LSP   Label Switched Path ([RFC3031])

LSR   Label Switching Router ([RFC3031])

MP2MP Multipoint to Multipoint

MPLS  Multiprotocol Label Switching ([RFC3031])

MPLS-TP  MPLS Transport Profile ([RFC5317])

Mb/s  Megabits per second (million bits per second)
NSP   Native Service Processing ([RFC3985])

NTP   Network Time Protocol

OAM   Operations, Administration, and Maintenance ([RFC6291])

OOB   Out-of-band (not carried within a data channel)

OTN   Optical Transport Network

P     Provider router ([RFC4364])

P2MP  Point to Multipoint

PE    Provider Edge router ([RFC4364])

PHB   Per-Hop Behavior ([RFC2475])

PHP   Penultimate Hop Popping ([RFC3443])

POS   PPP over SONET

PSC   This abbreviation has multiple interpretations.

      1.  Packet Switch Capable ([RFC3471]

      2.  PHB Scheduling Class ([RFC3270])

      3.  Protection State Coordination ([RFC6378])

PTP   Precision Time Protocol

PW    Pseudowire

QoS   Quality of Service

RA    Router Alert ([RFC3032])

RDI   Remote Defect Indication (MPLS-TP OAM)

RSVP-TE  RSVP Traffic Engineering

RTP   Real-time Transport Protocol

SCTP  Stream Control Transmission Protocol

SDH   Synchronous Data Hierarchy (European SONET, a form of TDM)
SONET Synchronous Optical Network (US SDH, a form of TDM)

T-LDP Targeted LDP (LDP sessions over more than one hop)

TC    Traffic Class ([RFC5462])

TCP   Transmission Control Protocol

TDM   Time-Division Multiplexing (legacy encapsulations)

TOS   Type of Service (see [RFC2474])

TTL   Time-to-live (a field in IP and MPLS headers)

UDP   User Datagram Protocol

UHP   Ultimate Hop Popping (opposite of PHP)

VCCV  Virtual Circuit Connectivity Verification ([RFC5085])

VLAN  Virtual Local Area Network (Ethernet)

VOQ   Virtual Output Queuing (switch fabric design)

VPN   Virtual Private Network

WG    Working Group

1.2. Use of Requirements Language 💅

okay so this document is Informational 🧚 The uppercase [RFC2119] key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" show up in this document in the following cases.

The RFC 2119 keywords appear where a requirement in this document is something a referenced RFC calls for. Usually the RFC that actually contains the requirement gets cited right inside the sentence, next to a keyword.
The keywords also appear where the document explicitly says they reflect operator experience that points to a requirement, even though no existing RFC requirement covers it.

Advice in this document may be ignored by implementations. Similarly, implementations that don't claim conformance to specific RFCs may ignore the requirements of those RFCs. In both cases, implementers should consider the risk of doing so ⚠️

1.3. Apparent Misconceptions 🙈

in the early days of forwarding silicon (which might be behind us now), there were apparently some misconceptions about MPLS. The following statements clear them up ✨

There are practical reasons to have more than one or two labels in an MPLS label stack. In some situations the label stack can get pretty deep. See Section 2.1.
Bestie, the label stack MUST be considered arbitrarily deep 💅 Section 3.27.4 ("Hierarchy: LSP Tunnels within LSPs") of RFC 3031 says "The label stack mechanism allows LSP tunneling to nest to any depth" [RFC3031]. If the bottom of the stack can't be found but there are enough labels to forward, an LSR MUST forward the packet. An LSR MUST NOT assume the packet is malformed unless the end of the packet is found before the bottom of the stack. See Section 2.1.
Deep label stacks are not rare in networks where they show up 🌈 Full packet rate performance is required regardless of label stack depth, except where multiple pop operations are required. See Section 2.1.
Research has shown that long bursts of short packets, with 40-byte or 44-byte IP payload sizes, are quite common. This comes from TCP ACK compression [ACK-compression]. The two sub-bullets below are advice that reflects very common nonnegotiable requirements of providers. Implementers may ignore this advice but should consider the risk of doing so 🫶 A. A forwarding engine SHOULD, if practical, be able to sustain an arbitrarily long sequence of small packets arriving at full interface rate. B. If indefinitely sustained full packet rate for small packets is not practical, a forwarding engine MUST be able to buffer a long sequence of small packets inbound to the on-chip decision engine and sustain full interface rate for some reasonable average packet rate. Without this small on-chip buffering, QoS-agnostic packet drops can occur.

See Section 2.3.

The implementations and system designs MUST support pseudowire Control Word (CW) if MPLS-TP is supported or if ACH [RFC5586] is being used on a pseudowire. The implementation and system designs SHOULD support pseudowire CW even if MPLS-TP and ACH [RFC5586] are not used. In that case they use CW and VCCV Type 1 [RFC5085] instead, which lets multipath work in the underlying network topology without messing with the PW traffic 🦋 [RFC7079] notes that there are still some deployments where the CW is not always used. It also notes that many service providers do enable the CW. See Section 2.4.1 for more discussion on why deployments SHOULD enable the pseudowire CW.

The following statements clarify more recent requirements that people often miss 👑

The implementer and system designer SHOULD support adding a pseudowire Flow Label [RFC6391]. Deployments MAY enable this feature for appropriate pseudowire types. See Section 2.4.3.
The implementer and system designer SHOULD support adding an MPLS Entropy Label [RFC6790]. Deployments MAY enable this feature. See Section 2.4.4.

Non-IETF definitions of MPLS exist, and these should not be used as normative texts in place of the relevant IETF RFCs. [RFC5704] documents incompatibilities between the IETF definition of MPLS and one such alternative MPLS definition, which led to significant issues in the resulting non-IETF specification 💀

1.4. Target Audience 🎯

this doc is for multiple audiences: the implementer (implementing MPLS forwarding in silicon or in software), the systems designer (putting together an MPLS forwarding system), and the deployer (running an MPLS network). These guidelines are meant to do the following things:

Explain what to do and what not to do when a deep label stack is encountered. (audience: implementer)
Highlight pitfalls to look for when implementing an MPLS forwarding chip. (audience: implementer)
Provide a checklist of features and performance specifications to request. (audience: systems designer, deployer)
Provide a set of tests to perform. (audience: systems designer, deployer).

the implementer, systems designer, and deployer have a transitive supplier-customer relationship. It's in the best interest of the supplier to review their product against their customer's checklist, and the secondary customer's checklist too if it applies 🍓

This document identifies and explains many details and potential pitfalls of MPLS forwarding. It's likely that the set of potential pitfalls identified here will later prove to be incomplete 🌸

2. Forwarding Issues ✨

A brief review of forwarding issues is provided in the subsections that follow. This section gives some background on why some of these requirements exist. The questions to ask suppliers are covered in Section 3. Some guidelines for testing are provided in Section 4 🧚

2.1. Forwarding Basics ✨

okay so the basic MPLS architecture and MPLS encapsulation (aka how packets actually get forwarded) are defined in [RFC3031] and [RFC3032] 📦. ngl those two are kinda LDP centric 💅. RSVP-TE is the one that handles traffic engineering (TE) and fast reroute, which are features LDP just doesn't have 🫶. The base document for MPLS RSVP-TE is [RFC3209].

A few RFCs update RFC 3032. The ones that impact forwarding are:

TTL processing is clarified in [RFC3443].
The use of MPLS Explicit NULL is modified in [RFC4182].
Differentiated Services is supported by [RFC3270] and [RFC4124]. The "EXP" field got renamed to "Traffic Class" in [RFC5462], which kills any misconception that it was just for experimenting or could be ignored 💀.
ECN is supported by [RFC5129].
The MPLS G-ACh and GAL are defined in [RFC5586].
[RFC5332] redefines the two data link layer codepoints for MPLS packets.

Tunneling encapsulations that carry MPLS, like MPLS in IP [RFC4023], MPLS in GRE [RFC4023], MPLS in L2TPv3 [RFC4817], or MPLS in UDP [MPLS-IN-UDP], are out of scope 🙅‍♀️.

Other RFCs have implications for MPLS Forwarding and do not update RFC 3032 or RFC 3209, including:

The pseudowire (PW) Associated Channel Header (ACH) is defined by [RFC5085] and was later generalized by the MPLS G-ACh [RFC5586].
The Entropy Label Indicator (ELI) and Entropy Label (EL) are defined by [RFC6790].

A few RFCs update RFC 3209. The ones listed as updating RFC 3209 generally only impact RSVP-TE signaling. Forwarding gets changed by the major extensions built on top of RFC 3209 🎀.

RFCs that impact forwarding get their own subsections down below 👇.

2.1.1. MPLS Special-Purpose Labels

okay so [RFC3032] says label values 0-15 are special-purpose labels with special meanings 🧚. [RFC7274] renamed these from the term "reserved labels" (used in [RFC3032]) to "special-purpose labels", which is a much better vibe ✨. There are three values of NULL label defined (two of which are later updated by [RFC4182]), and a Router Alert Label is defined too. The original intent was that special-purpose labels, except the NULL labels, could be sent to the routing engine CPU instead of being processed in the forwarding hardware. Hardware support is required by newer RFCs, like the ones defining Entropy Label and OAM that gets processed as a result of receiving a GAL. For new special-purpose labels, some accommodation is needed for LSRs that will send the labels to a general-purpose CPU or other highly programmable hardware 🤓. For example, ELI will only be sent to LSRs that have signaled support for [RFC6790], and a high OAM packet rate must be negotiated among endpoints.

[RFC3429] reserves a label for ITU-T Y.1711, but Y.1711 does not work with multipath, so its use is strongly discouraged 🙈.

The current list of special-purpose labels can be found on the "Multiprotocol Label Switching Architecture (MPLS) Label Values" registry, which lives on IANA's pages at http://www.iana.org 🌸.

[RFC7274] introduces an IANA "Extended Special-Purpose MPLS Label Values" registry and uses the "extension" label, label 15, to indicate that the next label is an extended special-purpose label and needs special handling 🔥. The range of only 16 values for special-purpose labels means a table can be used. The extended special-purpose labels have 20 bits available for use, so they may have to be handled some other way in the unlikely event that the currently reserved values 256-1048575 ever get used. If only the Standards Action range, 16-239, and the Experimental range, 240-255, are used, then a table of 256 entries can be used.

Unknown special-purpose labels and unknown extended special-purpose labels are handled the same way. When an unknown special-purpose label shows up, or a special-purpose label that isn't directly handled in forwarding hardware shows up, the packet should be sent to a general-purpose CPU by default. If this capability is supported, there must be an option to either drop or rate limit such packets based on the value of each special-purpose label 🫧.

2.1.2. MPLS Differentiated Services

[RFC2474] deprecates the IP Type of Service (TOS) and IP Precedence (Prec) fields and replaces them with the Differentiated Services Field, more commonly known as the Differentiated Services Code Point (DSCP) field 💜. [RFC2475] defines the Differentiated Services architecture, which in other forums is often called a Quality of Service (QoS) architecture.

MPLS uses the Traffic Class (TC) field to support Differentiated Services [RFC5462]. There are two primary documents describing how DSCP is mapped into TC:

[RFC3270] defines E-LSP and L-LSP. E-LSP uses a static mapping of DSCP into TC. L-LSP uses a per-LSP mapping of DSCP into TC, with one PHB Scheduling Class (PSC) per L-LSP. Each PSC can use multiple Per-Hop Behavior (PHB) values. For example, the Assured Forwarding service defines three PSCs, each with three PHBs [RFC2597].
[RFC4124] defines assignment of a class-type (CT) to an LSP, where a per-CT static mapping of TC to PHB is used. [RFC4124] provides a means to support up to eight E-LSP-like mappings of DSCP to TC.

To meet the Differentiated Services requirements specified in [RFC3270], the following forwarding requirements must be met. An ingress LER MUST be able to select an LSP and then apply a per-LSP map of DSCP into TC. A midpoint LSR MUST be able to apply a per-LSP map of TC to PHB. The number of mappings supported will be far less than the number of LSPs supported 🍓.

To meet the Differentiated Services requirements specified in [RFC4124], the following forwarding requirements must be met. An ingress LER MUST be able to select an LSP and then apply a per-LSP map of DSCP into TC. A midpoint LSR MUST be able to map LSP number to Class Type (CT), then use a per-CT map to map TC to PHB. Since there are only eight allowed values of CT, only eight maps of TC to PHB need to be supported. The LSP label can be used directly to find the TC-to-PHB mapping, which is what's needed to support L-LSPs as defined by [RFC3270] 🦋.

While support for [RFC4124] without [RFC3270] would only allow support for eight mappings of TC to PHB, it's common to support both and simply state a limit on the number of unique TC-to-PHB mappings that can be supported 💅.

2.1.3. Time Synchronization

PTP or NTP may be carried over MPLS [TIMING-OVER-MPLS]. Generally, NTP gets carried within IP, and IP gets carried in MPLS [RFC5905] ⏰. Both PTP and NTP benefit from accurate timestamping of incoming packets and the ability to insert accurate timestamps into outgoing packets. PTP correction that happens when forwarding requires updating a timestamp compensation field, based on the difference between packet arrival at an LSR and packet transmit time at that same LSR.

Since the label stack depth may vary, hardware should allow a timestamp to be placed in an outgoing packet at any specified byte position. It may be necessary to modify Layer 2 checksums or frame check sequences after insertion. PTP and NTP timestamp formats differ in a way that requires different implementations of the timestamp correction. If NTP or PTP is carried over UDP/IP or UDP/IP/MPLS, the UDP checksum will also have to be updated 🧩.

Accurate time synchronization is, in addition to being generally useful, required for MPLS-TP Delay Measurement (DM) OAM. See Section 2.6.4 🕰️.

2.1.4. Uses of Multiple Label Stack Entries

MPLS deployments in the early part of the prior decade (circa 2000) tended to support either LDP or RSVP-TE 📼. LDP was favored by some for its ability to scale to a very large number of PE devices at the edge of the network without adding deployment complexity. RSVP-TE was favored, generally in the network core, where traffic engineering and/or fast reroute were considered important.

Both LDP and RSVP-TE are used at the same time within major service provider networks, using a technique known as "LDP over RSVP-TE Tunneling" 🤝. This technique lets service providers carry LDP tunnels inside RSVP-TE tunnels. That makes it possible to take advantage of the traffic engineering and fast reroute on the more expensive intercity and intercontinental transport paths. The ingress RSVP-TE PE places many LDP tunnels on a single RSVP-TE LSP and carries it to the egress RSVP-TE PE. The LDP PEs are situated further from the core, for example within a metro network. LDP over RSVP-TE tunneling requires a minimum of two MPLS labels: one each for LDP and RSVP-TE.

The use of MPLS FRR [RFC4090] might add one more label to MPLS traffic, but only when FRR protection is in use (active). If LDP over RSVP-TE is in use and FRR protection is in use, then at least three MPLS labels are present on the label stack on the links the Bypass LSP traverses 🛡️. FRR is covered in Section 2.1.7.

LDP L2VPN, LDP IPVPN, BGP L2VPN, and BGP IPVPN added support for VPN services that the vast majority of service providers deploy 🌐. These VPN services add yet another label, bringing the label stack depth (when FRR is active) to four.

Pseudowires and VPN are discussed in further detail in Sections 2.1.8 and 2.1.9.

MPLS hierarchy, as described in [RFC4206] and updated by [RFC7074], can in principle add at least one additional label. MPLS hierarchy is discussed in Section 2.1.6.

Other features, like the Entropy Label (discussed in Section 2.4.4) and the Flow Label (discussed in Section 2.4.3), can add additional labels to the label stack 🏷️.

Although theoretical scenarios can easily result in eight or more labels, such cases are rare, if they occur at all today. For the purpose of forwarding, only the top label needs to be examined if PHP is used, and a few more if UHP is used (see Section 2.5). For deep label stacks, quite a few labels may have to be examined for the purpose of load balancing across parallel links (see Section 2.4). However, this depth can be bounded by a provider through use of the Entropy Label 🫶.

Other creative uses of MPLS within the IETF, such as using an MPLS label stack in source routing, may result in label stacks that are considerably deeper than those encountered today 🦄.

2.1.5. MPLS Link Bundling

MPLS Link Bundling was the first RFC to address the need for multiple parallel links between nodes [RFC4201] 🔗. MPLS Link Bundling is notable because it tried not to change MPLS forwarding, except in specifying the "all-ones" component link. MPLS Link Bundling is seldom, if ever, deployed. Instead, the multipath techniques described in Section 2.4 are used 🙃.

2.1.6. MPLS Hierarchy

MPLS hierarchy is defined in [RFC4206] and updated by [RFC7074] 🪆. Although RFC 4206 is considered part of GMPLS, the Packet Switching Capable (PSC) portion of the MPLS hierarchy applies to MPLS and may be supported in an otherwise GMPLS-free implementation. The MPLS PSC hierarchy remains the most likely means of providing further scaling in an RSVP-TE MPLS network, particularly where the network is designed to provide RSVP-TE connectivity to the edges. This is the case for the envisioned MPLS-TP networks. The use of the MPLS PSC hierarchy can add at least one additional label to a label stack, though it's likely that only one layer of PSC will be used in the near future 💖.

2.1.7. MPLS Fast Reroute (FRR) ✨

okay so fast reroute is defined by [RFC4090] 🦋 and it has two super different methods, the "One-to-One Backup" method, which uses the "Detour LSP", and the "Facility Backup", which uses a "bypass tunnel". people usually call these the detour and bypass methods, respectively 💖

the detour method uses a presignaled LSP. 🧚 hardware assistance may be needed for detour FRR so it can do local repair on a huge number of LSPs within the target of tens of milliseconds. ⚡ for every affected LSP, a swap operation has to be reprogrammed or switched over somehow. and ngl the detour method doubles the number of LSPs terminating at any given hop, and it bumps up the number of LSPs in the whole network by a factor that depends on the average detour path length 😭

the bypass method uses a tunnel that sits unused when nothing is broken, but it might carry lots of LSPs once a local repair is needed 🌸 there's no presignaling saying which working LSP gets diverted into which specific bypass LSP, so it's very much a "whoever shows up" situation 💅

if interface label space is used, the bypass LSP MUST extend one hop beyond the merge point, except if the merge point is the egress and PHP is used. 🎀

and if the bypass LSPs are not extended that way, then the merge LSR (the egress LSR of the bypass LSP) MUST use platform label space (as defined in [RFC3031]) 👑 so that an LSP working path on any given interface can be backed up by a bypass LSP ending on any other interface. the whole point is flexibility, bestie 🫶

hardware assistance may be needed here too, to do local repair on a large number of LSPs within tens of milliseconds ⏱️ for each affected LSP, a swap operation gets reprogrammed or switched over, with an extra push of the bypass LSP label. and the catch is that platform label space eats into the size of the LSR ILM for an LSR with a very large number of interfaces. 🙈

IP/LDP Fast Reroute (IP/LDP FRR) [RFC5714] is also applicable in MPLS networks 🍓 ECMP and Loop-Free Alternates (LFAs) [RFC5286] are the well-established IP/LDP FRR techniques, and they were the first ones to be widely deployed 🏆 work on IP/LDP FRR is still going on inside the IETF RTGWG 🌈 two big topics the RTGWG keeps talking about are microloops and partial coverage of those established techniques in some network topologies. [RFC5715] covers IP/LDP Fast Reroute microloops and microloop prevention 💜 RTGWG has also come up with extra IP/LDP FRR techniques to deal with the coverage concerns. they're extending LFA with remote LFA [REMOTE-LFA] 🦋 and there are other techniques that need new forwarding paths to be established, which are also under consideration. that includes the IPFRR "not-via" technique defined in [RFC6981] and maximally redundant trees (MRT) [MRT] ✨

here's the swap detail: ECMP, LFA (but not remote LFA), and MRT swap the top label to an alternate MPLS label. the other methods work similarly to the facility backup described in RFC 4090 and push an additional label. 🔥 IP/LDP FRR methods that push more than one label have been suggested, but they're still in early discussion 🧠

2.1.8. Pseudowire Encapsulation

the pseudowire (PW) architecture is defined in [RFC3985] 💖 when a pseudowire is carried over MPLS, it adds one or more additional label entries to the MPLS label stack. a PW Control Word is defined in [RFC4385], and the reasons for defining the Control Word are in [RFC4928] 🎀 the PW Associated Channel defined in [RFC4385] is used for OAM in [RFC5085] 🧚 and the PW Flow Label is defined in [RFC6391]. we'll talk more about it in this document in Section 2.4.3 ✨

there are loads of pseudowire encapsulations, supporting emulation of services like Frame Relay, ATM, Ethernet, TDM, and SONET/SDH over packet switched networks (PSNs) using IP or MPLS 🌸

the pseudowire encapsulation itself is out of scope for this document 🙈 but pseudowire impact on MPLS forwarding at the midpoint LSR is in scope 💅 so is the impact on ingress MPLS push and egress MPLS UHP pop. and while the encapsulation is out of scope, there's some advice on Sequence Number support 🫶

2.1.8.1. Pseudowire Sequence Number

Sequence Number support matters most for PW payload types that really expect lossless and/or in-order delivery 😭 identifying lost PW packets, and exactly how much payload was lost, is critical for PW services that maintain bit timing, like Time Division Multiplexing (TDM) services. these services MUST compensate lost payload on a bit-for-bit basis, so no shortcuts 💀

for PW services that maintain bit timing, packets that arrive out of order also MUST be identified 🔥 they MAY be either reordered or dropped. resequencing needs a "reorder buffer" in the egress PE, plus sequence numbering, and how well you can reorder is limited by how deep that buffer is. the down side of a big reorder buffer is more end-to-end service delay 🐢

for PW services that maintain bit timing, or any other service where jitter has to be bounded, a jitter buffer is always necessary 🌈 you need it whether or not reordering happens. and to be effective, a reorder buffer often has to be bigger than a jitter buffer needs to be, so you're basically trading off loss reduction against minimizing delay ⚖️

PW services that aren't timing-critical bit streams are cell oriented or frame oriented 🦋 resequencing support can be beneficial for cell- and frame-oriented payloads like ATM, FR, and Ethernet. it's desirable but not required 💜 honestly, the requirements for handling out-of-order packets vary a lot between services and deployments. for example, for an Ethernet PW, occasional (very rare) reordering is usually acceptable. and if the Ethernet PW is carrying MPLS-TP, then that reordering may be acceptable too 🍓

reducing jitter is best done by an end-system, since the tradeoff of loss vs. delay is different for each service 🧚 with interactive real-time services, low delay is preferred, but with non-interactive (one-way) real-time services, low loss is preferred. the same end-site might be receiving both kinds of traffic at once 👑 bounded jitter is sometimes a requirement for specific deployments, no matter what 💅

packet reordering should be rare, except in a small number of circumstances, and most of those come down to network design or equipment design errors 🙈

the most common case is when reordering is rare, happening only when a network or equipment fault forces traffic onto a new path with different delay. the packet loss that comes with a network or equipment fault is usually way more disruptive than any reordering that happens along the way 😭
a path change can happen for reasons other than a network or equipment fault, like an administrative routing change. this may cause packet reordering but generally without any packet loss ✨
if the edge isn't using pseudowire Control Word (CW) and the core is using multipath, reordering will be far more common. if that's happening, using CW on the edge will solve the problem 💖 without CW, resequencing isn't possible, because the Sequence Number lives in the CW 🎀
another avoidable case is when some core equipment has multipath and for some reason insists on periodically installing a new random number as the multipath hash seed 🤪 if supporting MPLS-TP, equipment MUST provide a means to disable periodic hash reseeding, and deployments MUST disable periodic hash reseeding. operator experience says that even if MPLS-TP isn't supported, equipment SHOULD provide a means to disable periodic hash reseeding, and deployments SHOULD disable periodic hash reseeding 🫶

in provider networks that use multipath techniques, and that may occasionally rebalance traffic or change PW paths for other reasons, reordering may be far more common than loss 🌊 where reordering is more common than loss, resequencing packets is beneficial, rather than dropping them at egress whenever out-of-order arrival happens. resequencing matters most for PW payload types that expect lossless delivery, since out-of-order delivery inside the network turns into PW loss in those cases 🔥

2.1.9. Layer 2 and Layer 3 VPN

Layer 2 VPN [RFC4664] and Layer 3 VPN [RFC4110] each add one or more label entries to the MPLS label stack 💜 the VPN encapsulations are out of scope for this document, but their impact on forwarding at the midpoint LSR is in scope 🦋

any of these services may be used on an ingress and egress that are MPLS Entropy Label enabled (see Section 2.4.4 for the discussion of Entropy Label) 🌸 that would add an extra two labels to the MPLS label stack. the need to provide a useful Entropy Label value affects the requirements of the VPN ingress LER, but that's out of scope for this document 🧚

2.2. MPLS Multicast

MPLS Multicast encapsulation is clarified in [RFC5332] ✨ MPLS Multicast may be signaled using RSVP-TE [RFC4875] or LDP [RFC6388] 🎀

[RFC4875] defines a root-initiated RSVP-TE LSP setup, instead of the leaf-initiated join used in IP multicast 🌈 [RFC6388] defines a leaf-initiated LDP setup. and both [RFC4875] and [RFC6388] define point-to-multipoint (P2MP) LSP setup. [RFC6388] also defined multipoint-to-multipoint (MP2MP) LSP setup 💅

the P2MP LSPs have a single source 👑 an LSR may be a leaf node, an intermediate node, or a "bud" node. a bud acts as both a leaf and an intermediate 🫶 at a leaf, an MPLS pop is performed, and the payload may be an IP multicast packet that needs further replication. at an intermediate node, an MPLS swap operation is performed. the bud needs both a pop operation and a swap operation performed on the same incoming packet, which is a lot of multitasking 🤹‍♀️

one strategy for P2MP functionality is to pop at the LSR interface serving as ingress to the P2MP traffic, and then optionally push labels at each LSR interface serving as egress to the P2MP traffic at that same LSR 🦋 a given LSR egress chip may support multiple egress interfaces, and each one needs a copy, but each with its own set of added labels and Layer 2 encapsulation. some physical interfaces have multiple sub-interfaces (like Ethernet VLAN or channelized interfaces), and each of those needs a copy too 🍓

if packet replication is performed at LSR ingress, then the ingress interface performance may suffer 😬 if replication happens inside an LSR switching fabric and at LSR egress, congestion of egress interfaces can't use backpressure to reach ingress interfaces with techniques like virtual output queuing (VOQ). if buffering is mainly supported at egress, the need for backpressure drops 💜 there may be no good solution for high volumes of multicast traffic if VOQ is used 💀

careful consideration should be given to the performance characteristics of high-fanout multicast for equipment intended to be used in that role 🔥

MP2MP LSPs are different, because any branch may provide an input, including a leaf 🌈 packets must be replicated onto all other branches. this forwarding is often implemented as multiple P2MP forwarding trees, one for each potential input interface at a given LSR ✨

2.3. Packet Rates ✨

okay so, the average packet size on the Internet can be pretty big 🐘, but long runs of tiny packets have been predicted in theory AND spotted in the wild 👀. Traffic compression and TCP ACK compression can team up to make long streams of packets with 40-44 bytes of payload 🙈. If those ride over Ethernet, the 64-byte minimum payload applies, which gives a packet rate of about 150 Mpps (million packets per second) for the whole burst on a nominal 100 Gb/s link 🔥. Other encapsulations can peak even higher, up to 250 Mpps (like when IP or MPLS gets wrapped in GFP over OTN ODU4) 💀.

It's possible that a specific implementation hits acceptable rates at a minimum payload size, like a 64-byte (64B) payload for Ethernet, but then the rate drops to an unacceptable level for other sizes, like a 65B payload 😭. And there are other packet rates worth caring about, not just TCP ACK. For instance, a TCP ACK carried over an Ethernet PW over MPLS over Ethernet might be 82B, or 82B plus an increment of 4B if extra MPLS labels are present 🎀.

If you graph packet rate vs. packet size, it usually looks like a sawtooth 🦋. That's commonly caused by a memory bottleneck and memory widths, sometimes an internal cache, but very often a super wide external buffer memory interface. In some cases it can come from a fabric transfer width instead. Fine packing, where you round up to the nearest 8B or 16B, gives a fine sawtooth with a small dip at 65B and an even smaller one at 82B 💅. Coarse packing, where you round up to 64B, can cause a sharper performance drop for 65B packets, or, maybe more importantly, a bigger drop for 82B packets 😬.

Losing some TCP ACK packets isn't the main worry when a burst like this hits. When one does, any other packet gets dropped once the on-chip input buffers in front of the decision engine are full, no matter its length or QoS 💔. Those buffers are often super tiny or don't exist at all (less than one packet of buffer), so you get significant QoS-agnostic packet drop 🙃.

Internet service providers and content providers once insisted on full rate forwarding with 40-byte payload packets as a requirement 👑. Today that requirement can often be waived, as long as the provider is convinced that no packets get dropped when long sequences of short packets show up 🫶.

Lots of equipment suppliers have pointed out that designing hardware that can process minimum size packets at full line rate costs a lot extra for very-high-speed interfaces 🥲. So if the hardware can't process minimum size packets at full line rate, then that hardware MUST be capable of handling large bursts of small packets, which is a situation that gets observed a lot 👀. This level of performance is needed to meet Differentiated Services [RFC2475] requirements. Without it, packets get lost before anyone can inspect the IP DSCP field [RFC2474] or MPLS TC field [RFC5462] 😭.

With enough on-chip buffers in front of the packet decision engine, an LSR can soak up a long run of short packets 🧚. Even if the output slows down enough that light congestion happens, the packets that already cleared the decision process can use the bigger VOQ or output side buffers and get handled per their configured QoS treatment, instead of being dropped completely at random ✨.

The buffering in front of the packet decision engine should be arranged so that 1) it can hold a relatively large number of small packets, 2) it can hold a small number of large packets, and 3) it can hold a mix of packets of different sizes 🌈.

These on-chip buffers don't need to add much delay, since they only kick in when the packet decision engine can't keep up, not in response to congestion, and they're pretty small anyway 🌸. For example, an on-chip buffer that can handle 4K packets of 64 bytes each, or 256KB, works out to 200 microseconds on a 10 Gb/s link and 20 microseconds on a 100 Gb/s link 🤯. If the packet decision engine can handle small packets at 90% of full rate, the maximum added delay is 20 microseconds and 2 microseconds respectively, and that delay only shows up if a 4K burst of short packets actually happens. When no burst of short packets is being processed, no delay gets added 💖. These buffers are only needed on high-speed interfaces where it's hard to process small packets at full line rate 🏎️.

Packet rate requirements apply no matter which network tier the equipment lives in, whether that's the network core or near the edges 🌍. If Differentiated Services requirements are going to be met, one of these two conditions MUST be met:

Packets must be processed at full line rate with minimum-sized packets. -OR-
Packets must be processed at a rate well under generally accepted average packet sizes, with sufficient buffering prior to the packet decision engine to accommodate long bursts of small packets.

2.4. MPLS Multipath Techniques 🛤️

In any large provider, whether service or content, hash-based multipath techniques get used in both the core and the edge 💜. In lots of these providers, hash-based multipath is also used in the bigger metro networks 🏙️.

There's a good reason for this: the Differentiated Services requirements say that packets within a common microflow SHOULD NOT be reordered [RFC2474] 🧠. Service providers usually go even stricter, commonly requiring that packets within a microflow MUST NOT be reordered, except in rare situations like load balancing across multiple links, a path change for load balancing, or a path change for some other reason 🔒.

The most common multipath techniques are ECMP applied at the IP forwarding level, Ethernet Link Aggregation Group (LAG) with inspection of the IP payload, and multipath on links carrying both IP and MPLS, where the IP header gets inspected below the MPLS label stack 🦄. In most core networks, the vast majority of traffic is MPLS encapsulated 💅.

To get a nicely balanced load across multiple links, IP header information has to be used 🎯. Common practice today is to reinspect the IP headers at each LSR and use the label stack and IP header information in a hash performed at each LSR. More details are in Section 2.4.5 🔍.

This technique is so everywhere in provider networks that any product without multipath support is basically unsuitable for large core networks 🚫. That's going to stay true for the near future, even as deployment of the MPLS Entropy Label starts to relax the core LSR multipath performance requirements. The reason is the existing deployed base of edge equipment that can't add an Entropy Label yet 🥺.

A generation of edge equipment that can add an MPLS Entropy Label is needed before the performance requirements for core LSRs can be relaxed. Still, it's likely that two generations of deployment down the road will let core LSRs support full packet rate only when a relatively small number of MPLS labels need inspecting before hashing 🔮. For now, don't count on it 🙅‍♀️.

Common practice today is to reinspect the packet at each LSR and use information from the packet combined with a hash seed selected by each LSR 🌀. Where Flow Labels or Entropy Labels are used, a hash seed must be used when creating these labels 🍓.

2.4.1. Pseudowire Control Word 🎀

Inside the core of a network, some form of multipath is almost certain to be used 🕸️. Multipath techniques deployed today are likely to look beneath the label stack for a chance to hash on IP addresses 👀.

A pseudowire encapsulated at a network edge must have a means to prevent reordering within the core, if the pseudowire crosses a network core or any part of a network topology where multipath is used (see [RFC4385] and [RFC4928]) 🛡️.

Not supporting the ability to encapsulate a pseudowire with a Control Word might lock a product out from consideration 🚪. A pseudowire capability without Control Word support might be fine for applications that are strictly both intra-metro and low bandwidth 🌷. But a provider with other applications will very likely not put up with equipment that can only support a subset of their pseudowire needs 😤.

2.4.2. Large Microflows 🐋

When multipath uses a simple hash and simple load balancing, like modulo or some other fixed allocation (see Section 2.4), there can be large microflows that each eat up 10% of the capacity of a component link of a potentially congested composite link 📉. One of those can upset the traffic balance, and more than one can drop the effective capacity of the entire composite link by more than 10% 😱.

When even a very small number of large microflows are around, there's a significant probability that more than one of them lands on the same component link 🎲. If the traffic contribution from large microflows is small, the probability of three or more large microflows on the same component link drops significantly 📊. So in a network with a significant number of parallel 10 Gb/s links, even a 1 Gb/s pseudowire or other large microflow that can't be subdivided into smaller flows should carry a Flow Label or Entropy Label if possible 🫧.

Actively managing the hash space to better accommodate large microflows has been implemented and deployed in the past. However, those techniques are out of scope for this document 🙏.

2.4.3. Pseudowire Flow Label 🦋

Unlike a pseudowire Control Word, a pseudowire Flow Label [RFC6391] is required only for pseudowires that have a relatively large capacity 💎. There are lots of cases where a pseudowire Flow Label makes sense. Any service, like a VPN carrying IP traffic within a pseudowire, can make use of one 💜.

Any pseudowire carried over MPLS that uses the pseudowire Control Word and does not carry a Flow Label is in effect a single microflow (in the terms defined in [RFC2475]), and it may run into the types of problems described in Section 2.4.2 😭.

2.4.4. MPLS Entropy Label ✨

okay so the MPLS Entropy Label makes flow group identification way easier at midpoint LSRs [RFC6790] 💖. Before this, midpoint LSRs had to dig through the entire label stack and often the IP headers 😭 just to spread traffic nicely across multipath stuff (see Section 2.4.5). With an Entropy Label, you can do the hash closer to the network edge, stick it in the label stack, and midpoint LSRs can just use it without fully re-inspecting the label stack or the payload ✨

The Entropy Label lets the core skip full label stack and payload inspection, which is exactly where performance is hardest to achieve (see Section 2.3) 🔥. The label stack inspection can stop as soon as the first Entropy Label shows up, which is usually after just a few labels get checked 🧚

To get these perks in the core, an LSR closer to the edge must be capable of adding an Entropy Label 🎀. This support may not be required in the access tier (the tier closest to the customer), but it's likely to be required at the edge or the border to the network core 🌸. An LSR peering with external networks will also need to be able to add an Entropy Label on incoming traffic 💅

2.4.5. Fields Used for Multipath Load Balance

the most common multipath techniques are based on a hash over a set of fields 🦋. Regardless of whether a hash is used or some other method is used, there's a limited set of fields that can safely be used for multipath 🫶

2.4.5.1. MPLS Fields in Multipath

if the "outer" or "first" layer of encapsulation is MPLS, then label stack entries get used in the hash 👑. Within a finite amount of time (and for small packets arriving at high speed, that time can be quite limited), only a finite number of label entries can be inspected 💀. Pipelined or parallel architectures improve this, but the limit is still finite, bestie 😭

the following guidelines are provided for using MPLS fields in multipath load balancing 📝

Only the 20-bit label field SHOULD be used. The TTL field SHOULD NOT be used. The S bit MUST NOT be used. The TC field (formerly EXP) MUST NOT be used. See text following this list for reasons 🙈
If an ELI label is found, then if the LSR supports Entropy Labels, the EL label field in the next label entry (the EL) SHOULD be used, label entries below that label SHOULD NOT be used, and the MPLS payload SHOULD NOT be used. See below this list for reasons 🍓
Special-purpose labels (label values 0-15) MUST NOT be used. Extended special-purpose labels (any label following label 15) MUST NOT be used. In particular, GAL and RA MUST NOT be used so that OAM traffic follows the same path as payload packets with the same label stack 🧚
If a new special-purpose label or extended special-purpose label is defined that requires special load-balance processing, then, just like the ELI label, a special action may be needed rather than skipping it 🔥
The most entropy is generally found in the label stack entries near the bottom of the label stack (innermost label, closest to S=1 bit) 💜. If the entire label stack can't be used (or the entire stack up to an EL), then it's better to use as many labels as possible closest to the bottom of the stack ✨
If no ELI is encountered, and the first nibble of the payload contains a 4 (IPv4) or 6 (IPv6), an implementation SHOULD support the ability to interpret the payload as IPv4 or IPv6 and extract and use appropriate fields from the IP headers 🌈. This feature is considered a nonnegotiable requirement by many service providers 😭. If supported, there MUST be a way to disable it (if, for example, PW without CW are used). This ability to disable this feature is considered a nonnegotiable requirement by many service providers too 💅

Therefore, an implementation has a very strong incentive to support both options 🫶

A label that is popped at egress (UHP pop) SHOULD NOT be used. A label that is popped at the penultimate hop (PHP pop) SHOULD be used ✨

Apparently, some chips have made use of the TC (formerly EXP) bits as a source of entropy, and this is very harmful 💀. It will reorder Assured Forwarding (AF) traffic [RFC2597] when a subset doesn't conform to the configured rates and gets remarked but not dropped at a prior LSR. Traffic that uses MPLS ECN [RFC5129] can also be reordered if TC is used for entropy 😱. So, as stated in the guidelines above, the TC field (formerly EXP) MUST NOT be used in multipath load balancing, because it violates Differentiated Services Ordered Aggregate (OA) requirements in these two instances 🎀

Use of the MPLS label entry S bit would put OAM traffic on a different path if adding a GAL at the bottom of the stack removed the S bit from the prior label 🙈

If an ELI label is found, then if the LSR supports Entropy Labels, the EL label field in the next label entry (the EL) SHOULD be used, and the search for additional entropy within the packet SHOULD be terminated 🦋. Failure to terminate the search will impact client MPLS-TP LSPs carried within server MPLS LSPs 😬. A network operator has the option to use administrative attributes to identify LSRs that don't terminate the entropy search at the first EL. Administrative attributes are defined in [RFC3209]. Some configuration is required to support this 🔧

If the label removed by a PHP pop is not used, then for any PW for which CW is used, there's no basis for a multipath load split 💔. In some networks it's infeasible to put all PW traffic on one component link. Any PW that doesn't use CW will be improperly split, regardless of whether the label removed by a PHP pop is used. So the PHP pop label SHOULD be used, as recommended above 🌸

2.4.5.2. IP Fields in Multipath

inspecting the IP payload provides the most entropy in provider networks 🌟. The practice of looking past the bottom of stack label for an IP payload is well accepted and documented in [RFC4928] and in other RFCs 📚

where IP is mentioned in the document, both IPv4 and IPv6 apply 🦋. All LSRs MUST fully support IPv6 👑

when information in the IP header is used, the following guidelines apply 💖

Both the IP source address and IP destination address SHOULD be used. There MAY be an option to reverse the order of these addresses, which improves the ability to provide symmetric paths in some cases. Many service providers require that both addresses be used 🫶
Implementations SHOULD allow inspection of the IP protocol field and use of the UDP or TCP port numbers. For many service providers this feature is considered mandatory, especially for enterprise, data center, or edge equipment 💅. If this feature is provided, it SHOULD be possible to disable use of TCP and UDP ports. Many service providers consider it a nonnegotiable requirement that use of UDP and TCP ports can be disabled 😭

Therefore, there's a strong incentive for implementations to provide both options 🎀

Equipment suppliers MUST NOT make assumptions that because the IP version field is equal to 4 (an IPv4 packet), the IP protocol will either be TCP (IP protocol 6) or UDP (IP protocol 17) and blindly fetch the data at the offset where the TCP or UDP ports would be found 🙅‍♀️. With IPv6, TCP and UDP port numbers are not at fixed offsets. With IPv4 packets carrying IP options, TCP and UDP port numbers are not at fixed offsets either 💀
The IPv6 header flow field SHOULD be used. This is the explicit purpose of the IPv6 flow field; however, observed flow fields rarely contain a non-zero value 🥲. Some uses of the flow field have been defined, such as [RFC6438]. In the absence of MPLS encapsulation, the IPv6 flow field can serve a role equivalent to the Entropy Label ✨
Support for other protocols that share a common Layer 4 header, such as RTP [RFC3550], UDP-Lite [RFC3828], SCTP [RFC4960], and DCCP [RFC4340], SHOULD be provided, especially for edge or access equipment where additional entropy may be needed 🧚. Equipment SHOULD also use RTP, UDP-lite, SCTP, and DCCP headers when creating an Entropy Label 🌈
The following IP header fields should not or must not be used:
- A. Similar to avoiding TC in MPLS, the IP DSCP and ECN bits MUST NOT be used 🚫
- B. The IPv4 TTL or IPv6 Hop Count SHOULD NOT be used 🙈
- C. Note that the IP TOS field was deprecated. ([RFC0791] was updated by [RFC2474].) No part of the IP DSCP field can be used (formerly IP PREC and IP TOS bits) 📝
Some IP encapsulations support tunneling, such as IP-in-IP, GRE, L2TPv3, and IPsec. These provide a greater source of entropy that some provider networks carrying large amounts of tunneled traffic may need, for example as used in [RFC5640] for GRE and L2TPv3 🌊. The use of tunneling header information is out of scope for this document 🙃

This document makes the following recommendations. These recommendations are not required to claim compliance to any existing RFC, so implementers are free to ignore them. But due to service provider requirements, they should consider the risk of doing so ⚠️. The use of IP addresses MUST be supported, and TCP and UDP ports (conditional on the protocol field and properly located) MUST be supported. The ability to disable use of UDP and TCP ports MUST be available 🎀

Though potentially very useful in some networks, it's uncommon to support using payloads of tunneling protocols carried over IP 🤔. Though the use of tunneling protocol header information is out of scope for this document, it is not discouraged 🦋

2.4.5.3. Fields Used in Flow Label

the ingress to a pseudowire (PW) can extract information from the payload being encapsulated to create a Flow Label 🧁. [RFC6391] references IP carried in Ethernet as an example. The Native Service Processing (NSP) function defined in [RFC3985] differs with pseudowire type 💜. It's in the NSP function where information for a specific type of PW can be extracted for use in a Flow Label. Determining which fields to use for any given PW NSP is out of scope for this document 🌸

2.4.5.4. Fields Used in Entropy Label ✨

okay so the Entropy Label gets slapped on at the ingress to an LSP 🦋 The payload being wrapped is usually MPLS, a PW, or IP, and you can tell which one it is from the Layer 2 encapsulation (Ethernet, GFP, POS, etc.) 💅

if the payload is MPLS, the info used to make the Entropy Label is the same info used for local load balancing (see Section 2.4.5.1) 🫶 and this info MUST be extracted to generate the Entropy Label even if the LSR's local egress interface is not a multipath 😭 that part is non-negotiable bestie

for the non-MPLS payload types, only the ones that actually get forwarded are of interest 🌸 ARP payloads aren't forwarded, so they're not the vibe here. CLNP payloads, which are only used for IS-IS, are also not forwarded 💀

the non-MPLS payload types of greatest interest are IPv4 and IPv6 🌈 the guidelines in Section 2.4.5.2 apply to the fields used to create an Entropy Label, so go check that section too ✨

also, the IP tunneling protocols mentioned in Section 2.4.5.2 may be more applicable when generating an Entropy Label at the edge or access 🧚 why? because deep packet inspection is practical there thanks to lower interface speeds, whereas in the core it may be impractical 👑

2.5. MPLS-TP and UHP 🎀

okay this section is giving drama 😭 MPLS-TP brings forwarding demands that will be extremely difficult to meet in a core network 💀 the most troublesome one is the requirement for Ultimate Hop Popping (UHP), which is the opposite of Penultimate Hop Popping (PHP) 🔥

using UHP opens the door to one or more MPLS pop operations plus an MPLS swap operation for each packet 🍓 that means the potential for multiple lookups and multiple counter instances per packet is real, bestie 🙈

as networks grow and LDP LSPs get tunneled into RSVP-TE LSPs, and/or RSVP-TE hierarchy gets used, the requirement to do one or more MPLS pop operations plus an MPLS swap operation (and possibly a push or two) only gets bigger 📈 and if MPLS-TP LM (link monitoring) OAM is enabled at each layer, then a packet and byte count MUST be maintained for each pop and swap operation so OAM can be offered for each layer 💜

2.6. Local Delivery of Packets 🌸

there are a bunch of situations where packets are destined for a local address or where a return packet has to be generated 🦋 the risk here is an outage, whether from attacks on network infrastructure or from accidental misconfiguration that overloads the processor 😭 so some hardware assistance is needed for all traffic headed to the general-purpose CPU that handles the MPLS control protocol or the network management protocol, and in most cases for other general-purpose CPUs on an LSR too 🔥 why? because it's super easy to overwhelm that processor with traffic coming in on high-speed interfaces, whether the traffic is malicious or not 💀

DoS protection is an area that needs hardware support and is often overlooked or not thought through enough 🙈 hardware assists are also needed for OAM, especially the more demanding MPLS-TP OAM 🧚

2.6.1. DoS Protection 🛡️

modern equipment supports a whole lot of control-plane and management-plane protocols 💅 generally, no single way of protecting network gear from DoS attacks is enough, especially for high-speed interfaces 🔥 this isn't specific to MPLS, but it's something you can't ignore when you're implementing or evaluating MPLS implementations 👑

two types of protections get cited as the main ways to guard against attacks of all kinds ✨

Isolated Control/Management Traffic control and management traffic can be carried out-of-band (OOB), meaning not mixed in with payload 🫶 for MPLS, using G-ACh and GAL to carry control and management traffic gives you isolation from potentially malicious payloads 🦋 but if that's the only thing you rely on, compromising a single node (even a small computer at a network operations center) could compromise an entire network 😭 implementations that send all G-ACh/GAL traffic directly to a routing engine CPU are open to DoS attacks as a result of that kind of compromise 💀
Cryptographic Authentication cryptographic authentication can be super effective at preventing malicious injection of control or management traffic 💜 but it can, in some circumstances, be hit with a DoS attack that overwhelms the decryption capacity with a high volume of malicious traffic 🙈 for very-low-speed interfaces, cryptographic authentication can be performed by the general-purpose CPU used as a routing engine 🌈 for all other cases, cryptographic hardware may be needed 🔥 and for very-high-speed interfaces, even cryptographic hardware can get overwhelmed 💀

some control and management protocols are often carried along with payload traffic 🌸 this is commonly the case with BGP, T-LDP, and SNMP, and it's often the case with RSVP-TE too 🧚 even when they're carried over G-ACh/GAL, extra measures can reduce the chance that a minor breach gets leveraged into a full network attack 👑

some of the additional protections are supported by hardware packet filtering ✨

GTSM [RFC5082] defines a mechanism that uses the IPv4 TTL or IPv6 Hop Limit fields to make sure control traffic that can only originate from an immediate neighbor isn't forged and isn't coming from a distant source 🫶 GTSM can be applied to many control protocols that are routable, for example LDP [RFC6720] 🦋
IP Filtering at the very minimum, packet filtering plus classification and the use of multiple queues that support rate limiting is needed for traffic that could potentially be sent to a general-purpose CPU used as a routing engine 😭 the first level of filtering only allows connections to be initiated from specific IP prefixes to specific destination ports, and then preferably passes traffic directly to a cryptographic engine and/or rate limits it 🔥 the second level of filtering passes connected traffic, such as TCP connections that have received at least one authenticated SYN or that were locally initiated 💅 this second level only passes traffic to specific address and port pairs so they can be checked for cryptographic authentication 💜

cryptographic authentication is generally the last resort in DoS attack mitigation 🚨 if a packet has to go to a general-purpose CPU first and then to a cryptographic engine, a DoS attack is possible on high-speed interfaces 💀 only where hardware can fully process a cryptographic authentication without a general-purpose CPU getting involved (to find the authentication field and to identify the portion of the packet the cryptographic algorithm runs over) is cryptographic authentication beneficial in protecting against DoS attacks 👑

for chips supporting multiple 100 Gb/s interfaces, only a very large number of parallel cryptographic engines can provide the processing capacity to handle a large-scale DoS or distributed DoS (DDoS) attack 😭 for many forwarding chips, that much processing power takes up significant chip real estate and power, which reduces system space and power density 🙈 so for this reason, cryptographic authentication is not considered a viable first line of defense 🚫

for some networks, the first line of defense is some means of supporting OOB control and management traffic 🌈 in the past, that OOB channel might have used overhead bits in SONET or OTN, or a dedicated DWDM wavelength 🧚 G-ACh and GAL provide an alternative OOB mechanism that is independent of underlying layers ✨ in other networks, including most IP/MPLS networks, perimeter filtering serves a similar purpose, though it's less effective without extreme vigilance 👀

a second line of defense is filtering, including GTSM 🛡️ for protocols such as EBGP, GTSM and other filtering are often the first line of defense 🫶 cryptographic authentication is usually the last line of defense and isn't enough by itself to mitigate DoS or DDoS attacks 💜

2.6.2. MPLS OAM 🔍

[RFC4377] defines requirements for MPLS OAM that predate MPLS-TP 🕰️ [RFC4379] defines what's commonly called LSP Ping and LSP Traceroute 🦋 [RFC4379] is updated by [RFC6424], which supports MPLS tunnels and stitched LSP and P2MP LSP 🫶 [RFC4379] is also updated by [RFC6425], which supports P2MP LSP 💅 and [RFC4379] is updated by [RFC6426] to support MPLS-TP connectivity verification (CV) and route tracing ✨

[RFC4950] extends the ICMP format to support TTL expiration that may occur when using IP Traceroute within an MPLS tunnel 🧚 the ICMP message generation can be implemented in forwarding hardware, but if the ICMP packets are sent to a general-purpose CPU, this packet flow must be rate limited to avoid a potential DoS attack 🛑

[RFC5880] defines Bidirectional Forwarding Detection (BFD), a protocol intended to detect faults in the bidirectional path between two forwarding engines 🌈 [RFC5884] and [RFC5885] define BFD for MPLS 💜 BFD can provide failure detection on any kind of path between systems, including direct physical links, virtual circuits, tunnels, MPLS Label Switched Paths (LSPs), multihop routed paths, and unidirectional links as long as there is some return path 🙌

the processing requirements for BFD are lower than for LSP Ping, which makes BFD somewhat better suited for relatively high-rate proactive monitoring ⚡ BFD does not verify that the data plane matches the control plane, but LSP Ping does 👀 LSP Ping is somewhat better suited for on-demand monitoring, including relatively low-rate periodic verification of the data plane, and as a diagnostic tool 🔎

hardware assistance is often provided for BFD response where BFD setup or parameter change isn't involved, and it may be necessary for relatively high-rate proactive monitoring 🔥 if both BFD and LSP Ping are recognized in filtering before traffic is passed to a general-purpose CPU, appropriate DoS protection can be applied (see Section 2.6.1) 🛡️ failing to recognize BFD and LSP Ping, and at least rate limit them, creates the potential for misconfiguration to cause outages rather than just errors in the misconfigured OAM 😭

2.6.3. Pseudowire OAM 🎀

pseudowire OAM makes use of the control channel provided by Virtual Circuit Connectivity Verification (VCCV) [RFC5085] 🦋 VCCV makes use of the pseudowire Control Word 💅 BFD support over VCCV is defined by [RFC5885], and [RFC5885] is updated by [RFC6478] in support of static pseudowires 🫶 [RFC4379] is updated by [RFC6829] to support LSP Ping for Pseudowire FEC advertised over IPv6 ✨

G-ACh/GAL (defined in [RFC5586]) is the preferred MPLS-TP OAM control channel and applies to any MPLS-TP endpoints, including pseudowire 👑 see Section 2.6.4 for an overview of MPLS-TP OAM 🧚

2.6.4. MPLS-TP OAM 🦋

okay so [RFC6669] is basically the cheat sheet for the MPLS-TP OAM toolset 📖✨. It's the collection of protocols that actually supports the MPLS-TP OAM requirements from [RFC5860] and fits into the MPLS-TP OAM framework from [RFC6371]. main character energy 💅

Here's the toolset, bestie:

CC-CV: [RFC6428] defines BFD extensions for proactive Continuity Check and Connectivity Verification (CC-CV) applications 🌸. [RFC6426] adds LSP Ping extensions for on-demand connectivity verification.
RDI: Remote Defect Indication (RDI) gets triggered when proactive CC-CV, which is BFD based, fails 🚨. For fast RDI, RDI SHOULD be initiated and handled by hardware if BFD is handled in forwarding hardware. [RFC6428] gives BFD an extension that includes RDI in the BFD format and explains how the indication is supposed to be used 🫶.
Route Tracing: [RFC6426] says the LSP Ping enhancements for MPLS-TP on-demand connectivity verification also cover using LSP Ping for route tracing of an MPLS-TP path 🗺️.
Alarm Reporting: [RFC6427] describes a new protocol for Alarm Indication Signal (AIS), Link Down Indication (LDI), and fault management 🔔. ngl this one is serious. If forwarding hardware can't support it, you could miss protection recovery time requirements, so support of this functionality is strongly recommended 😭.
Lock Instruct: Lock instruct is initiated on demand, so it doesn't need to be implemented in forwarding hardware 🔒. [RFC6435] defines a lock instruct protocol.
Lock Reporting: [RFC6427] covers lock reporting 📣. It doesn't need to be implemented in forwarding hardware.
Diagnostic: [RFC6435] defines protocol support for loopback 🔁. Loopback initiation is on demand, so it doesn't need to be implemented in forwarding hardware. Loopback of packet traffic SHOULD be implemented in forwarding hardware on high-speed interfaces 🔥.
Packet Loss and Delay Measurement: [RFC6374] and [RFC6375] define a protocol and profile for Packet Loss Measurement (LM) and Delay Measurement (DM) 📊. LM needs very accurate capture and insertion of packet and byte counters when a packet is transmitted, plus capture of those counters when a packet is received. This capture and insertion MUST be implemented in forwarding hardware for LM OAM if high accuracy is needed. DM needs very accurate capture and insertion of a timestamp on transmission, plus capture of the timestamp when a packet is received. This timestamp capture and insertion MUST be implemented in forwarding hardware for DM OAM if high accuracy is needed ⏱️💀.

See Section 2.6.2 for the discussion of hardware support needed for BFD and LSP Ping 🧚.

CC-CV and alarm reporting are tied to protection, so they SHOULD be supported in forwarding hardware to protect a large number of affected LSPs within target response intervals 🛡️. With MPLS-TP, CC-CV runs on BFD, so hardware help for BFD processing helps make sure protection recovery time requirements can be met, even when faults hit a huge number of LSPs 💖.

MPLS-TP Protection State Coordination (PSC) is defined by [RFC6378] and updated by [RFC7324], which fixes some errors in [RFC6378] 🩹✨.

2.6.5. MPLS OAM and Layer 2 OAM Interworking 🎀

[RFC6670] explains why we picked a single MPLS-TP OAM solution 🙌. It also looks at the consequences if ITU-T had made a second OAM solution based on Ethernet encodings and mechanisms. iykyk, it's a whole thing 🙈.

[RFC6310] and [RFC7023] specify how defect states map between lots of hardware Attachment Circuits (ACs) and their pseudowires (PWs) 🔗. This functionality SHOULD be supported in forwarding hardware.

It's beneficial if an MPLS OAM implementation can interwork with the underlying server layer and also give a way to interwork with a client layer 🌈. For example, [RFC6427] specifies inter-layer propagation of AIS and LDI from an MPLS server layer to client MPLS layers. When the server layer uses a Layer 2 mechanism, like Ethernet, PPP over SONET/SDH, or GFP over OTN, interworking among layers is beneficial too 🍓. On high-speed interfaces, supporting this interworking in forwarding hardware helps make sure protection based on it can meet recovery time requirements, even when faults hit a large number of LSPs 🏎️.

2.6.6. Extent of OAM Support by Hardware 🧠

Sometimes certain requirements must be met, like relatively high CC-CV rates with a large number of interfaces, or strict protection recovery time requirements with a moderate number of affected LSPs. In those cases, some OAM functionality must be supported by forwarding hardware 💪. In other cases, like highly accurate LM and DM OAM, or strict protection recovery time requirements with a large number of affected LSPs, OAM functionality must be entirely implemented in forwarding hardware 😤.

Where possible, put the implementation in programmable forwarding hardware. If standards change or get extended later, that way the changes can probably be handled with a hardware reprogram instead of a whole replacement 🔄✨.

For some functionality there's a strong case for dedicated forwarding hardware. Examples include the packet and byte counters needed for LM OAM, and the ones needed for management protocols 📦. Likewise, the capture and insertion of packet and byte counts or timestamps for transmitted LM or DM or time synchronization packets MUST be implemented in forwarding hardware if high accuracy is required ⏰.

For some functions there's a strong case for limited support in forwarding hardware, but an external general-purpose processor may be used if performance criteria can be met 🧩. For example, origination of RDI triggered by CC-CV, response to RDI, and Protection State Coordination (PSC) functionality may be supported by hardware. Expanding to a large number of client LSPs, and sending AIS or RDI to client LSPs, may happen on a general-purpose processor instead. Some forwarding hardware has one or more on-chip general-purpose processors that are well suited for this role 🤖. [RFC7324] is a very recent document that affects a protection state machine needing hardware support, which really underlines how important some programmability in forwarding hardware is 🥹.

The customer (system supplier or provider) should not dictate design, but should independently validate target functionality and performance 🧐. Still, it's not uncommon for service providers and system implementers to insist on reviewing design details (under a non-disclosure agreement). That's because of past experiences with suppliers, and they reject suppliers who won't share details 🙅‍♀️.

2.6.7. Support for IPFIX in Hardware 📈

The IPFIX architecture is defined by [RFC5470]. IPFIX supports per-flow statistics 📊. The IPFIX information elements (IEs) are defined in [RFC7012] and include IEs for MPLS 🏷️.

The forwarding chips used in core routers aren't optimized for high-touch apps like IPFIX 🙃. So support for IPFIX in core routers is often limited to optional IPFIX metering. That involves 1-in-N packet sampling, limited filtering support, and redirection to either an internal CPU or an external interface. The CPU or device at the other end of the external interface then does the full IPFIX filtering and IPFIX collector functionality 🛰️.

LSRs deployed further from the core may support lower-capacity interfaces, but they can handle higher-touch apps on the forwarding hardware. They may also have dedicated hardware that covers a bigger subset of IPFIX functionality before handing off to a general-purpose CPU ✨. In some cases, far from the core, the entire IPFIX functionality (up to and including the collector) may be built into the hardware and firmware of the forwarding silicon 🦾. It's also worth noting that at lower speeds, a general-purpose CPU may be good enough to run IPFIX, especially if metering is used 🫧.

2.7. Number and Size of Flows 🌊

Service provider networks may carry up to hundreds of millions of flows on 10 Gb/s links 🤯. Most flows are super short-lived, many under a second ⚡. A subset of the flows are low capacity and somewhat long-lived. When Internet traffic dominates capacity, a very small subset of flows are high capacity and/or very long-lived 👑.

Two types of limitations around the number and size of flows have been observed 👀:

Some hardware can't handle some high-capacity flows because of internal paths that are limited, like per-packet backplane paths, or paths inside or outside chips such as buffer memory paths 🚧. Such designs can handle aggregates of smaller flows. Some hardware with acknowledged limitations has been successfully deployed, but it may get more problematic if the capacity of large microflows in deployed networks keeps growing 📉.
Some hardware approaches can't handle a large number of flows, or a large number of large flows, because they try to count per flow instead of dealing with aggregates of flows 🫠. Hash techniques scale with the number of flows because of a fixed hash size, though many flows end up in the same hash bucket. Techniques that identify individual flows have been implemented but have never been successfully deployed for Internet traffic 🚫.

3. Questions for Suppliers 📝

The following questions should be asked of a supplier 💌. These questions are grouped into broad categories and are meant to be open-ended questions to the supplier. The tests in Section 4 are meant to verify whether the supplier disclosed any compliance or performance limitations completely and accurately 🔍✨.

3.1. Basic Compliance 💖

Q#1 okay so, can the implementation forward packets with an arbitrarily large stack depth? 🥹 If there are limits, what are they, and when do extra limits kick in? (Like high packet rate, specific features being turned on, or specific types of packet processing.) See Section 2.1.

Q#2 Is the entire set of basic MPLS functionality described in Section 2.1 supported? 💅

Q#3 Is the set of MPLS special-purpose labels handled correctly and with adequate performance? And are extended special-purpose labels handled correctly and with adequate performance? See Section 2.1.1. ✨

Q#4 Are mappings of label value and TC to PHB handled correctly? That includes L-LSP mappings (RFC 3270) and CT mappings (RFC 4124) to PHB. See Section 2.1.2. 🎀

Q#5 Is time synchronization adequately supported in forwarding hardware? 🕰️

A. Are both PTP and NTP formats supported?
B. Is the accuracy of timestamp insertion and incoming stamping sufficient?

See Section 2.1.3.

Q#6 Is link bundling supported? 🔗

A. Can an LSP be pinned to specific components?
B. Is the "all-ones" component link supported?

See Section 2.1.5.

Q#7 Is MPLS hierarchy supported? 🪆

A. Are both PHP and UHP supported? What limitations exist on the number of pop operations with UHP?
B. Are the pipe, short-pipe, and uniform models supported? Are TTL and TC values updated correctly at egress where applicable?

See Section 2.1.6 regarding MPLS hierarchy. See [RFC3443] regarding PHP, UHP, and pipe, short-pipe, and uniform models. 🦋

Q#8 Is FRR supported? 🛡️

A. Are both "One-to-One Backup" and "Facility Backup" supported?
B. What forms of IP/LDP FRR are supported?
C. How quickly does protection recovery occur?
D. Does protection recovery speed increase when a fault affects a large number of protected LSPs? And if so, by how much?

See Section 2.1.7.

Q#9 Are pseudowire Sequence Numbers handled correctly? See Section 2.1.8.1. 🧩

Q#10 Is VPN LER functionality handled correctly and without performance issues? See Section 2.1.9. 👑

Q#11 Is MPLS multicast (P2MP and MP2MP) handled correctly? 📡

A. Are packets dropped on uncongested outputs if some outputs are congested?
B. Is performance limited in high-fanout situations?

See Section 2.2.

3.2. Basic Performance 🚀

Q#12 Can very small packets be forwarded at full line rate on all interfaces indefinitely? What limitations exist? And under what circumstances do further limitations come into play? (Such as specific features enabled or specific types of packet processing.) 🐜

Q#13 Customers must decide whether to relax the prior requirement and to what extent. If the answer to the prior question indicates that limitations exist, then:

A. What is the smallest packet size where full line rate forwarding can be supported?
B. What is the longest burst of full-rate small packets that can be supported?

Specify circumstances (such as specific features enabled or specific types of packet processing) that often impact these rates and burst sizes. 📊

Q#14 How many pop operations can be supported along with a swap operation at full line rate, while maintaining per-LSP packet and byte counts for each pop and swap? This requirement is particularly relevant for MPLS-TP. 🎯

Q#15 How many label push operations can be supported. While this limitation is rarely an issue, it applies to both PHP and UHP, unlike the pop limit that applies to UHP. 🌸

Q#16 For a worst case where all packets arrive on one LSP, what is the counter overflow time? Are any means provided to avoid polling all counters at short intervals? This applies to both MPLS and MPLS-TP. ⏱️

3.3. Multipath Capabilities and Performance 🛣️

Multipath capabilities and performance do not apply to MPLS-TP, but they apply to MPLS and apply if MPLS-TP is carried in MPLS. 💜

Q#17 How are large microflows accommodated? Is there active management of the hash space mapping to output ports? See Section 2.4.2. 🌊

Q#18 How many MPLS labels can be included in a hash based on the MPLS label stack? 🏷️

Q#19 Is packet rate performance decreased beyond some number of labels? 📉

Q#20 Can the IP header and payload information below the MPLS stack be used in the hash? If so, which IP fields, payload types, and payload fields are supported? 🔍

Q#21 At what maximum MPLS label stack depth can Bottom of Stack and an IP header appear without impacting packet rate performance? 📏

Q#22 Are special-purpose labels excluded from the label stack hash? Are extended special-purpose labels excluded from the label stack hash? See Section 2.4.5.1. 🙈

Q#23 How is multipath performance affected by high-capacity flows, an extremely large number of flows, or very short-lived flows? See Section 2.7. 🌪️

3.4. Pseudowire Capabilities and Performance 🧵

Q#24 Is the pseudowire Control Word supported? 🎀

Q#25 What is the maximum rate of pseudowire encapsulation and decapsulation? Apply the same questions as in Section 3.2 ("Basic Performance") for any packet-based pseudowire, such as IP VPN or Ethernet. 💅

Q#26 Does inclusion of a pseudowire Control Word impact performance? 🤔

Q#27 Are Flow Labels supported? 🏳️

Q#28 If so, what fields are hashed on for the Flow Label for different types of pseudowires? 🧾

Q#29 Does inclusion of a Flow Label impact performance? ⚡

3.5. Entropy Label Support and Performance 🌈

Q#30 Can an Entropy Label be added when acting as an ingress LER, and can it be removed when acting as an egress LER? 🍓

Q#31 If an Entropy Label can be added, what fields are hashed on for the Entropy Label? 🥄

Q#32 Does adding or removing an Entropy Label impact packet rate performance? 🏎️

Q#33 Can an Entropy Label be detected in the label stack, used in the hash, and properly terminate the search for further information to hash on? 🛑

Q#34 Does using an Entropy Label have any negative impact on performance? It should have no impact or a positive impact. ✨

3.6. DoS Protection 🛡️

Q#35 For each control- and management-plane protocol in use, what measures are taken to provide DoS attack hardening? 🏰

Q#36 Have DoS attack tests been performed? 🧪

Q#37 Can compromise of an internal computer on a management subnet be leveraged for any form of attack, including DoS attack? 😱

3.7. OAM Capabilities and Performance 🔧

Q#38 What OAM proactive and on-demand mechanisms are supported? 🛠️

Q#39 What performance limits exist under high proactive monitoring rates? 📈

Q#40 Can excessively high proactive monitoring rates impact control-plane performance or cause control-plane instability? 💀

Q#41 Ask the prior questions for each of the following. 🫶

A. MPLS OAM
B. Pseudowire OAM
C. MPLS-TP OAM
D. Layer 2 OAM Interworking

See Section 2.6.

4. Forwarding Compliance and Performance Testing 🧪

Okay, bestie, real talk: packet rate performance for equipment with a whole bunch of 10 Gb/s or 100 Gb/s links is not possible using desktop computers or workstations. 😭 Using high-end workstations as a source of test traffic was barely viable 20 years ago, and it is no longer at all viable. Custom microcode has been used on specialized router forwarding cards to generate test traffic and measure it, but for the most part, performance testing will require specialized test equipment. There are multiple sources of suitable equipment. 🛒

The tests listed here do not correspond one-to-one to the set of questions in Section 3. The same categorization is used, and these tests largely serve to validate answers provided to the prior questions. They can also provide answers where a supplier is unwilling to disclose compliance or performance. 🕵️

Performance testing is the domain of the IETF Benchmark Methodology Working Group (BMWG). 🎓 Below are brief descriptions of conformance and performance tests. Some very basic tests, specified in [RFC5695], partially cover only the basic performance test T#3.

The following tests should be performed by the systems designer or deployer; or, if it is not practical for the potential customer to perform the tests directly, they may be performed by the supplier on their behalf. These tests are grouped into broad categories. 🗂️

The tests in Section 4.1 should be repeated under various conditions to retest basic performance when critical capabilities are enabled. Repeating the full set of performance tests for every capability, and every combination of capabilities, would be very time intensive. Therefore, a reduced set of performance tests can be used to gauge the impact of enabling specific capabilities. 💫

4.1. Basic Compliance 💖

T#1 Test forwarding at a high rate for packets with a varying number of label entries. While packets with more than a dozen label entries are unlikely to be used in any practical scenario today, it is useful to know if limitations exists. 🧐

T#2 For each of the questions listed under "Basic Compliance" in Section 3, verify the claimed compliance. For any functionality considered critical to a deployment, the applicable performance using each capability under load should be verified in addition to basic compliance. 🎀

4.2. Basic Performance 💅

T#3 okay so, test packet forwarding at full line rate with small packets 🔥 See [RFC5695]. The most likely case to fail is the smallest packet size. Also test with packet sizes in 4-byte increments, ranging from payload sizes of 40 to 128 bytes. bestie this is the stress test era 💀

T#4 if the prior tests did not succeed for all packet sizes, then do the following tests 🫶

A. Increase the packet size by 4 bytes until you find a size that can be forwarded at full rate.
B. Inject bursts of consecutive small packets into a stream of larger packets. Allow some time for recovery between bursts. Keep increasing the number of packets in the burst until packets get dropped. 😭

T#5 send test traffic where a swap operation is required 🦋 Also set up multiple LSPs carried over other LSPs, where the device under test (DUT) is the egress of these LSPs. Create test packets so the swap operation is performed after pop operations, and keep increasing the number of pop operations until forwarding of small packets at full line rate can no longer be supported. Also check how many pop operations can be supported before the full set of counters can no longer be maintained. This one is especially relevant for MPLS-TP, so it's giving main character energy 👑

T#6 send all traffic on one LSP and see if the counters become inaccurate 🙈 Often, counters on silicon are much smaller than the 64-bit packet and byte counters in various IETF MIBs. System developers should consider what counter polling rate is necessary to maintain accurate counters, and whether those polling rates are practical. Relevant MIBs for MPLS are discussed in [RFC4221] and [RFC6639].

T#7 [RFC6894] provides a good basis for MPLS FRR testing ✨ Similar testing should be performed to determine restoration times. However, this testing is far more difficult to perform, because you need a simulated test topology that can simulate the signaling used in restoration. The simulated topology should be comparable with the target deployment in the number of nodes and links, and in resource usage, flooding and setup delays. Some commercial test equipment can support this type of testing 🧚

4.3. Multipath Capabilities and Performance 🌈

Multipath capabilities do not apply to MPLS-TP, but they do apply to MPLS, and they apply if MPLS-TP is carried in MPLS. Not me sorting out which layer is which 😭

T#8 send traffic at a rate well exceeding the capacity of a single multipath component link, where entropy exists only below the top of the stack 💖 If only the top label is used, this test will fail immediately. rip 💀

T#9 move the labels with entropy down in the stack until either the full forwarding rate can no longer be supported, or most or all packets try to use the same component link 🍓

T#10 repeat the two tests above with the entropy contained in IP headers or IP payload fields below the label stack, rather than in the label stack 🦋 Test with the set of IP headers or IP payload fields considered relevant to the deployment or to the target market.

T#11 determine whether traffic that contains a pseudowire Control Word is interpreted as IP traffic 🎀 Information in the payload MUST NOT be used in the load balancing if the first nibble of the packet is not 4 or 6 (IPv4 or IPv6). The MUST NOT here is non-negotiable, bestie 🫶

T#12 determine whether special-purpose labels and extended special-purpose labels are excluded from the label stack hash 💅 They MUST be excluded. Iconic behaviour, no notes.

T#13 perform testing in the presence of combinations of 🌸

A. Very large microflows.
B. Relatively short-lived high-capacity flows.
C. Extremely large numbers of flows.
D. Very short-lived small flows.

4.4. Pseudowire Capabilities and Performance 🧚

T#14 ensure that a pseudowire can be set up with a pseudowire label and pseudowire Control Word added at ingress, and the pseudowire label and pseudowire Control Word removed at egress ✨

T#15 for a pseudowire that contains variable-length payload packets, repeat the performance tests listed under "Basic Performance" for pseudowire ingress and egress functions 💜

T#16 repeat pseudowire performance tests with and without a pseudowire Control Word 🙈

T#17 determine whether a pseudowire can be set up with a pseudowire label, Flow Label, and pseudowire Control Word added at ingress, and the pseudowire label, Flow Label, and pseudowire Control Word removed at egress 🌈

T#18 determine which payload fields are used to create the Flow Label, and whether the set of fields and algorithm provide sufficient entropy for load balancing 🔥

T#19 repeat pseudowire performance tests with Flow Labels included. Lowkey the most fun one so far 🦋

4.5. Entropy Label Support and Performance 🎀

T#20 determine whether Entropy Labels can be added at ingress and removed at egress 💖

T#21 determine which fields are used to create an Entropy Label. Labels further down in the stack, including Entropy Labels further down and IP headers or IP payload fields where applicable, should be used. Determine whether the set of fields and algorithm provide sufficient entropy for load balancing 🧠

T#22 repeat the performance tests under "Basic Performance" when Entropy Labels are used, where ingress or egress is the device under test (DUT) 🍓

T#23 determine whether an ELI is detected when acting as a midpoint LSR, and whether the search for further information on which to base the load balancing is used. Information below the Entropy Label SHOULD NOT be used. Don't go digging where you're not supposed to, bestie 🙈

T#24 ensure that the Entropy Label indicator and Entropy Label (ELI and EL) are removed from the label stack during UHP and PHP operations 🌸

T#25 ensure that operations on the TC field, when adding and removing an Entropy Label, are correctly carried out 💅 If TC is changed during a swap operation, the ability to transfer that change MUST be provided. The ability to suppress the transfer of TC MUST also be provided. See the pipe, short-pipe, and uniform models in [RFC3443]. Two MUSTs in one test, it's giving serious spec energy 👑

T#26 repeat the performance tests for a midpoint LSR with Entropy Labels found at various label stack depths ✨

4.6. DoS Protection 🛡️

T#27 actively attack LSRs under high protocol churn load, and determine the control-plane performance impact or a successful DoS under test conditions 😭 Specifically, test for the following.

A. TCP SYN attack against control-plane and management-plane protocols using TCP, including CLI access (typically SSH-protected login), NETCONF, etc.
B. High traffic volume attack against control-plane and management-plane protocols not using TCP.
C. Attacks that can be performed from a compromised management subnet computer, but not one with authentication keys.
D. Attacks that can be performed from a compromised peer within the control plane (internal domain and external domain). Assume that keys that are per peering and keys that are per router ID, rather than network-wide keys, are in use.

See Section 2.6.1 🫶

4.7. OAM Capabilities and Performance 🔍

T#28 determine maximum sustainable rates of BFD traffic 🌸 If BFD requires CPU intervention, determine both maximum rates and CPU loading when multiple interfaces are active.

T#29 verify LSP Ping and LSP Traceroute capability ✨

T#30 determine maximum rates of MPLS-TP CC-CV traffic 💖 If CC-CV requires CPU intervention, determine both maximum rates and CPU loading when multiple interfaces are active.

T#31 determine MPLS-TP DM precision 🦋

T#32 determine MPLS-TP LM accuracy 🧚

T#33 verify MPLS-TP AIS/RDI and Protection State Coordination (PSC) functionality, protection speed, and AIS/RDI notification speed when a large number of Maintenance Entities (MEs) must be notified with AIS/RDI. That's a lot of notifications, periodt 🔥

5. Security Considerations 🔒✨

okay bestie, this doc is basically a review. it just looks at forwarding behavior that's specified somewhere else and points out the compliance and performance requirements, so it introduces no new security requirements or concerns. no cap 💅

Hardware support and other equipment hardening against DoS attacks are covered in Section 2.6.1. Section 3.6 has a list of DoS questions to ask suppliers 📝. Section 4.6 suggests kinds of testing that can give you some assurance that a supplier's DoS hardening claims are actually true 🧪.

Knowing about potential performance shortcomings can help new implementations dodge pitfalls 🦋. It's unlikely that this knowledge could be the basis of a new denial of service, since these pitfalls are already widely known in the service provider community and among leading equipment suppliers. In practice, you'd need extreme data and packet rates to affect existing equipment, and to affect networks that are still vulnerable because they never added adequate protection. Those extreme rates make this type of denial of service unlikely, and make undetectable denial of service of this type impossible 🙅‍♀️.

Every normative reference has its own security considerations 📚. Here's a quick summary of the MPLS security considerations that matter for forwarding:

MPLS encapsulation doesn't support an authentication extension 🙃. This is reflected in the security section of [RFC3032]. Documents that clarify MPLS header fields, like TTL [RFC3443], the explicit null label [RFC4182], renaming EXP to TC [RFC5462], ECN for MPLS [RFC5129], and MPLS Ethernet encapsulation [RFC5332], make no changes to the security considerations in [RFC3032].
Some of the cited RFCs are about Diffserv forwarding 🎀. [RFC3270] talks about MPLS and Diffserv security. [RFC2474] mentions theft of service and denial of service due to mismarking. It also mentions IPsec interaction, but since MPLS isn't carried by IP, that kind of interaction from [RFC2474] doesn't apply here.
[RFC3209] is only cited here because of its make-before-break forwarding requirements 🔁. This is tied to resource sharing, so the theft-of-service and denial-of-service concerns in [RFC2474] still apply.
[RFC4090] defines FRR, which gives you protection without adding any security concerns 🛡️. RFC 4201 defines link bundling, and it doesn't raise any extra security concerns either.
Various OAM control channels are defined in [RFC4385] (PW CW), [RFC5085] (VCCV), and [RFC5586] (G-Ach and GAL) 👀. These documents describe potential abuse of those OAM control channels.
[RFC4950] defines ICMP extensions for when the MPLS TTL expires and the payload is IP 📨. This gives you MPLS header information, which is no use to an IP attacker, but sending it can be switched off through configuration.
GTSM [RFC5082] helps protect against high-volume traffic spoofing, which is a form of DoS attack 🌊.
BFD [RFC5880] [RFC5884] [RFC5885] is a form of OAM used in MPLS and MPLS-TP 💓. The security considerations for the OAM control channel are relevant. The BFD payload supports authentication. The MPLS encapsulation, the MPLS control channel, or the PW control channel that BFD may be carried in do not support authentication 😬. When an IP return OAM path is used, IPsec is suggested for securing that return path.
Other forms of OAM are supported by [RFC6374] [RFC6375] (Loss and Delay Measurement), [RFC6428] (Continuity Check/Verification based on BFD), and [RFC6427] (Fault Management) 🧠. The security considerations for the OAM control channel apply here too. If IP return paths are used, they can be secured with IPsec.
Linear protection is defined by [RFC6378] and updated by [RFC7324] 🔀. Security concerns for MPLS encapsulation and OAM control channels apply. The security concerns also restate [RFC5920] as it applies to protection switching.
The PW Flow Label [RFC6391] and the MPLS Entropy Label [RFC6790] both affect multipath load balancing ⚖️. The security concerns restate [RFC5920]. The security impact is limited to load distribution.

MPLS security, including data-plane security, gets a deeper dive in [RFC5920] (MPLS/GMPLS Security Framework) 🔍. The MPLS-TP security framework [RFC6941] builds on that, focusing mostly on the MPLS-TP OAM additions and OAM channels, with some attention to using network management in place of control-plane setup. In both security framework documents, MPLS is assumed to run within a "trusted zone", which is defined as the area where a single service provider has total operational control over that part of the network 🏰.

If control-plane security and management-plane security are robust enough, then compromising a single network element can cause chaos in the data plane anywhere in the network through denial-of-service attacks 💥. It won't turn into a Byzantine security failure where other network elements are fully compromised 🙌.

MPLS security, or the lack of it, can affect whether traffic gets misrouted and lost, intercepted, intercepted and reinserted (a man-in-the-middle attack), or spoofed 🕵️‍♀️. End-user applications, including the control-plane and management-plane protocols the service provider uses, are expected to use appropriate end-to-end authentication and, where it makes sense, end-to-end encryption 🔐.

⚠️ bestie, the big takeaway: MPLS security depends on the network being a trusted zone, and the layers above it need their own end-to-end protection. don't skip that part 😭.

6. Organization of References Section 🗂️

The References section is split into Normative and Informative subsections 💕. References that directly specify forwarding encapsulations or behaviors are listed as normative. References that describe only signaling are listed as informative. They're normative for signaling, but they're informative with respect to MPLS forwarding 🧁.

7. References 📖

7.1. Normative References 💜

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC3032]  Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y.,
           Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack
           Encoding", RFC 3032, January 2001.

[RFC3209]  Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
           and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
           Tunnels", RFC 3209, December 2001.

[RFC3270]  Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen,
           P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-
           Protocol Label Switching (MPLS) Support of Differentiated
           Services", RFC 3270, May 2002.

[RFC3443]  Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing
           in Multi-Protocol Label Switching (MPLS) Networks", RFC
           3443, January 2003.

[RFC4090]  Pan, P., Swallow, G., and A. Atlas, "Fast Reroute
           Extensions to RSVP-TE for LSP Tunnels", RFC 4090, May
           2005.

[RFC4182]  Rosen, E., "Removing a Restriction on the use of MPLS
           Explicit NULL", RFC 4182, September 2005.

[RFC4201]  Kompella, K., Rekhter, Y., and L. Berger, "Link Bundling
           in MPLS Traffic Engineering (TE)", RFC 4201, October 2005.

[RFC4385]  Bryant, S., Swallow, G., Martini, L., and D. McPherson,
           "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for
           Use over an MPLS PSN", RFC 4385, February 2006.

[RFC4950]  Bonica, R., Gan, D., Tappan, D., and C. Pignataro, "ICMP
           Extensions for Multiprotocol Label Switching", RFC 4950,
           August 2007.

[RFC5082]  Gill, V., Heasley, J., Meyer, D., Savola, P., and C.
           Pignataro, "The Generalized TTL Security Mechanism
           (GTSM)", RFC 5082, October 2007.

[RFC5085]  Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit
           Connectivity Verification (VCCV): A Control Channel for
           Pseudowires", RFC 5085, December 2007.

[RFC5129]  Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion
           Marking in MPLS", RFC 5129, January 2008.

[RFC5332]  Eckert, T., Rosen, E., Aggarwal, R., and Y. Rekhter, "MPLS
           Multicast Encapsulations", RFC 5332, August 2008.

[RFC5586]  Bocci, M., Vigoureux, M., and S. Bryant, "MPLS Generic
           Associated Channel", RFC 5586, June 2009.

[RFC5880]  Katz, D. and D. Ward, "Bidirectional Forwarding Detection
           (BFD)", RFC 5880, June 2010.

[RFC5884]  Aggarwal, R., Kompella, K., Nadeau, T., and G. Swallow,
           "Bidirectional Forwarding Detection (BFD) for MPLS Label
           Switched Paths (LSPs)", RFC 5884, June 2010.

[RFC5885]  Nadeau, T. and C. Pignataro, "Bidirectional Forwarding
           Detection (BFD) for the Pseudowire Virtual Circuit
           Connectivity Verification (VCCV)", RFC 5885, June 2010.

[RFC6374]  Frost, D. and S. Bryant, "Packet Loss and Delay
           Measurement for MPLS Networks", RFC 6374, September 2011.

[RFC6375]  Frost, D. and S. Bryant, "A Packet Loss and Delay
           Measurement Profile for MPLS-Based Transport Networks",
           RFC 6375, September 2011.

[RFC6378]  Weingarten, Y., Bryant, S., Osborne, E., Sprecher, N., and
           A. Fulignoli, "MPLS Transport Profile (MPLS-TP) Linear
           Protection", RFC 6378, October 2011.

[RFC6391]  Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan,
           J., and S. Amante, "Flow-Aware Transport of Pseudowires
           over an MPLS Packet Switched Network", RFC 6391, November
           2011.

[RFC6427]  Swallow, G., Fulignoli, A., Vigoureux, M., Boutros, S.,
           and D. Ward, "MPLS Fault Management Operations,
           Administration, and Maintenance (OAM)", RFC 6427, November
           2011.

[RFC6428]  Allan, D., Swallow Ed. , G., and J. Drake Ed. , "Proactive
           Connectivity Verification, Continuity Check, and Remote
           Defect Indication for the MPLS Transport Profile", RFC
           6428, November 2011.

[RFC6790]  Kompella, K., Drake, J., Amante, S., Henderickx, W., and
           L. Yong, "The Use of Entropy Labels in MPLS Forwarding",
           RFC 6790, November 2012.

[RFC7324]  Osborne, E., "Updates to MPLS Transport Profile Linear
           Protection", RFC 7324, July 2014.

7.2. Informative References 🌈

[ACK-compression]
           Zhang, L., Shenker, S., and D. Clark, "Observations and
           Dynamics of a Congestion Control Algorithm: The Effects of
           Two-Way Traffic", Proc. ACM SIGCOMM, ACM Computer
           Communications Review (CCR) Vol. 21, No. 4, pp. 133-147.,
           1991.

[MPLS-IN-UDP]
           Xu, X., Sheth, N., Yong, L., Pignataro, C., and F.
           Yongbing, "Encapsulating MPLS in UDP", Work in Progress,
           January 2014.

[MRT]      Atlas, A., Kebler, R., Bowers, C., Envedi, G., Csaszar,
           A., Tantsura, J., Konstantynowicz, M., and R. White, "An
           Architecture for IP/LDP Fast-Reroute Using Maximally
           Redundant Trees", Work in Progress, July 2014.

[REMOTE-LFA]
           Bryant, S., Filsfils, C., Previdi, S., Shand, M., and S.
           Ning, "Remote LFA FRR", Work in Progress, May 2014.

[RFC0791]  Postel, J., "Internet Protocol", STD 5, RFC 791, September
           1981.

Last stretch of the bibliography, bestie 📚✨ these informative refs are giving main character energy 💅

[RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998.

[RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998.

[RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, "Assured Forwarding PHB Group", RFC 2597, June 1999.

[RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, January 2001.

[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, September 2001.

[RFC3429] Ohta, H., "Assignment of the 'OAM Alert Label' for Multiprotocol Label Switching Architecture (MPLS) Operation and Maintenance (OAM) Functions", RFC 3429, November 2002.

[RFC3471] Berger, L., "Generalized Multi-Protocol Label Switching (GMPLS) Signaling Functional Description", RFC 3471, January 2003.

[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.

[RFC3828] Larzon, L-A., Degermark, M., Pink, S., Jonsson, L-E., and G. Fairhurst, "The Lightweight User Datagram Protocol (UDP-Lite)", RFC 3828, July 2004.

[RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to-Edge (PWE3) Architecture", RFC 3985, March 2005.

[RFC4023] Worster, T., Rekhter, Y., and E. Rosen, "Encapsulating MPLS in IP or Generic Routing Encapsulation (GRE)", RFC 4023, March 2005.

[RFC4110] Callon, R. and M. Suzuki, "A Framework for Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs)", RFC 4110, July 2005.

[RFC4124] Le Faucheur, F., "Protocol Extensions for Support of Diffserv-aware MPLS Traffic Engineering", RFC 4124, June 2005.

[RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) Hierarchy with Generalized Multi-Protocol Label Switching (GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005.

[RFC4221] Nadeau, T., Srinivasan, C., and A. Farrel, "Multiprotocol Label Switching (MPLS) Management Overview", RFC 4221, November 2005.

[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, March 2006.

[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, February 2006.

[RFC4377] Nadeau, T., Morrow, M., Swallow, G., Allan, D., and S. Matsushima, "Operations and Management (OAM) Requirements for Multi-Protocol Label Switched (MPLS) Networks", RFC 4377, February 2006.

[RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol Label Switched (MPLS) Data Plane Failures", RFC 4379, February 2006.

[RFC4664] Andersson, L. and E. Rosen, "Framework for Layer 2 Virtual Private Networks (L2VPNs)", RFC 4664, September 2006.

[RFC4817] Townsley, M., Pignataro, C., Wainner, S., Seely, T., and J. Young, "Encapsulation of MPLS over Layer 2 Tunneling Protocol Version 3", RFC 4817, March 2007.

[RFC4875] Aggarwal, R., Papadimitriou, D., and S. Yasukawa, "Extensions to Resource Reservation Protocol - Traffic Engineering (RSVP-TE) for Point-to-Multipoint TE Label Switched Paths (LSPs)", RFC 4875, May 2007.

[RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal Cost Multipath Treatment in MPLS Networks", BCP 128, RFC 4928, June 2007.

[RFC4960] Stewart, R., "Stream Control Transmission Protocol", RFC 4960, September 2007.

[RFC5036] Andersson, L., Minei, I., and B. Thomas, "LDP Specification", RFC 5036, October 2007.

[RFC5286] Atlas, A. and A. Zinin, "Basic Specification for IP Fast Reroute: Loop-Free Alternates", RFC 5286, September 2008.

[RFC5317] Bryant, S. and L. Andersson, "Joint Working Team (JWT) Report on MPLS Architectural Considerations for a Transport Profile", RFC 5317, February 2009.

[RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic Class" Field", RFC 5462, February 2009.

[RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, "Architecture for IP Flow Information Export", RFC 5470, March 2009.

[RFC5640] Filsfils, C., Mohapatra, P., and C. Pignataro, "Load-Balancing for Mesh Softwires", RFC 5640, August 2009.

[RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding Benchmarking Methodology for IP Flows", RFC 5695, November 2009.

[RFC5704] Bryant, S., Morrow, M., and IAB, "Uncoordinated Protocol Development Considered Harmful", RFC 5704, November 2009.

[RFC5714] Shand, M. and S. Bryant, "IP Fast Reroute Framework", RFC 5714, January 2010.

[RFC5715] Shand, M. and S. Bryant, "A Framework for Loop-Free Convergence", RFC 5715, January 2010.

[RFC5860] Vigoureux, M., Ward, D., and M. Betts, "Requirements for Operations, Administration, and Maintenance (OAM) in MPLS Transport Networks", RFC 5860, May 2010.

[RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithms Specification", RFC 5905, June 2010.

[RFC5920] Fang, L., "Security Framework for MPLS and GMPLS Networks", RFC 5920, July 2010.

[RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, D., and S. Mansfield, "Guidelines for the Use of the "OAM" Acronym in the IETF", BCP 161, RFC 6291, June 2011.

[RFC6310] Aissaoui, M., Busschbach, P., Martini, L., Morrow, M., Nadeau, T., and Y(J). Stein, "Pseudowire (PW) Operations, Administration, and Maintenance (OAM) Message Mapping", RFC 6310, July 2011.

[RFC6371] Busi, I. and D. Allan, "Operations, Administration, and Maintenance Framework for MPLS-Based Transport Networks", RFC 6371, September 2011.

[RFC6388] Wijnands, IJ., Minei, I., Kompella, K., and B. Thomas, "Label Distribution Protocol Extensions for Point-to-Multipoint and Multipoint-to-Multipoint Label Switched Paths", RFC 6388, November 2011.

[RFC6424] Bahadur, N., Kompella, K., and G. Swallow, "Mechanism for Performing Label Switched Path Ping (LSP Ping) over MPLS Tunnels", RFC 6424, November 2011.

[RFC6425] Saxena, S., Swallow, G., Ali, Z., Farrel, A., Yasukawa, S., and T. Nadeau, "Detecting Data-Plane Failures in Point-to-Multipoint MPLS - Extensions to LSP Ping", RFC 6425, November 2011.

[RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS On-Demand Connectivity Verification and Route Tracing", RFC 6426, November 2011.

[RFC6435] Boutros, S., Sivabalan, S., Aggarwal, R., Vigoureux, M., and X. Dai, "MPLS Transport Profile Lock Instruct and Loopback Functions", RFC 6435, November 2011.

[RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label for Equal Cost Multipath Routing and Link Aggregation in Tunnels", RFC 6438, November 2011.

[RFC6478] Martini, L., Swallow, G., Heron, G., and M. Bocci, "Pseudowire Status for Static Pseudowires", RFC 6478, May 2012.

[RFC6639] King, D. and M. Venkatesan, "Multiprotocol Label Switching Transport Profile (MPLS-TP) MIB-Based Management Overview", RFC 6639, June 2012.

[RFC6669] Sprecher, N. and L. Fang, "An Overview of the Operations, Administration, and Maintenance (OAM) Toolset for MPLS-Based Transport Networks", RFC 6669, July 2012.

[RFC6670] Sprecher, N. and KY. Hong, "The Reasons for Selecting a Single Solution for MPLS Transport Profile (MPLS-TP) Operations, Administration, and Maintenance (OAM)", RFC 6670, July 2012.

[RFC6720] Pignataro, C. and R. Asati, "The Generalized TTL Security Mechanism (GTSM) for the Label Distribution Protocol (LDP)", RFC 6720, August 2012.

[RFC6829] Chen, M., Pan, P., Pignataro, C., and R. Asati, "Label Switched Path (LSP) Ping for Pseudowire Forwarding Equivalence Classes (FECs) Advertised over IPv6", RFC 6829, January 2013.

[RFC6894] Papneja, R., Vapiwala, S., Karthik, J., Poretsky, S., Rao, S., and JL. Le Roux, "Methodology for Benchmarking MPLS Traffic Engineered (MPLS-TE) Fast Reroute Protection", RFC 6894, March 2013.

[RFC6941] Fang, L., Niven-Jenkins, B., Mansfield, S., and R. Graveman, "MPLS Transport Profile (MPLS-TP) Security Framework", RFC 6941, April 2013.

[RFC6981] Bryant, S., Previdi, S., and M. Shand, "A Framework for IP and MPLS Fast Reroute Using Not-Via Addresses", RFC 6981, August 2013.

[RFC7012] Claise, B. and B. Trammell, "Information Model for IP Flow Information Export (IPFIX)", RFC 7012, September 2013.

[RFC7023] Mohan, D., Bitar, N., Sajassi, A., DeLord, S., Niger, P., and R. Qiu, "MPLS and Ethernet Operations, Administration, and Maintenance (OAM) Interworking", RFC 7023, October 2013.

[RFC7074] Berger, L. and J. Meuric, "Revised Definition of the GMPLS Switching Capability and Type Fields", RFC 7074, November 2013.

[RFC7079] Del Regno, N. and A. Malis, "The Pseudowire (PW) and Virtual Circuit Connectivity Verification (VCCV) Implementation Survey Results", RFC 7079, November 2013.

[RFC7274] Kompella, K., Andersson, L., and A. Farrel, "Allocating and Retiring Special-Purpose MPLS Labels", RFC 7274, June 2014.

[TIMING-OVER-MPLS] Davari, S., Oren, A., Bhatia, M., Roberts, P., and L. Montini, "Transporting Timing messages over MPLS Networks", Work in Progress, April 2014.

Appendix A. Acknowledgements 💖

okay so this is the thank-you section, and bestie, the list of people who helped is iconic 🫶 Numerous very useful comments came in through private email, and some of those contributions are credited here, roughly in chronological order.

Paul Doolan gave a quick review that led to a bunch of clarifications 🔥, most notably about on-chip vs. system buffering, the 100 Gb/s link speed assumptions behind the 150 Mpps figure, and how large microflows get handled. Pablo Frank pointed out the sawtooth effect in PPS vs. packet-size graphs 📈, which is why a few paragraphs on that got added. Comments from Lou Berger at IETF 85 are the reason Section 2.7 exists 🦋.

Valuable comments came in on the BMWG mailing list too 💌. Jay Karthik flagged some testing methodology hints. After discussion they were judged out of scope and removed, but they might help future BMWG work 🧚.

Nabil Bitar pointed out that QoS (Differentiated Services), MPLS multicast (P2MP and MP2MP), and MPLS-TP OAM needed coverage 💅. Nabil also clarified a bunch of questions and tests in Sections 3 and 4 🎀.

Mark Szczesniak gave a thorough review with lots of useful comments and suggestions that made the document better ⭐.

Gregory Mirsky and Thomas Beckhaus offered useful comments during the MPLS Review Team review 🌸.

Tal Mizrahi's comments led to clarifications about timestamp processing, local delivery of packets, and the need for hardware assistance when processing OAM traffic 💜.

Alexander (Sasha) Vainshtein found errors in Section 2.1.8.1 😭 and suggested new text. After a long discussion, that text restated the summary of requirements from PWE3 RFCs and more clearly described the benefits and drawbacks of packet resequencing based on the PW Sequence Number 🧚.

Loa Anderson gave useful comments and corrections before WGLC. Adrian Farrel gave useful comments and corrections as part of the AD review 🌈.

Discussion with Steve Kent during SecDir review expanded Section 5, briefly summarizing security considerations related to forwarding in normative references 🔐. Tom Petch pointed out some editorial errors in private email plus an important math error 🙈. Al Morton during OpsDir review prompted clarification in the section about the target audience, suggested clearer wording in places, and found numerous editorial errors 🍓.

Discussion with Stewart Bryant and Alia Atlas as part of IESG review resulted in coverage of IPFIX and improvements to document coverage of MPLS FRR, and IP/LDP FRR, plus some corrections elsewhere 👑.

Authors' Addresses 🎀

Here are the authors' details, exactly as they appear in the RFC 💌

Curtis Villamizar (editor)
Outer Cape Cod Network Consulting, LLC

EMail: curtis@occnc.com


Kireeti Kompella
Juniper Networks

EMail: kireeti@juniper.net


Shane Amante
Apple Inc.
1 Infinite Loop
Cupertino, California  95014

EMail: amante@apple.com


Andrew Malis
Huawei Technologies

EMail: agmalis@gmail.com


Carlos Pignataro
Cisco Systems
7200-12 Kit Creek Road
Research Triangle Park, NC  27709
US

EMail: cpignata@cisco.com

📦 where this page comes from

ietf: rfc7325 2014-08
💖 girlified by claude-haiku-5-5 on 2026-10-18: a language model's rewrite of the original, under its terms