RFC 5129

Explicit Congestion Marking in MPLS

📅 published January 2008, from ietf

Explicit Congestion Marking in MPLS 🎀

Okay so this is RFC 5129 🦋, from the Network Working Group, and it's called Explicit Congestion Marking in MPLS. The authors are B. Davie (Cisco Systems, Inc.), B. Briscoe, and J. Tay (BT Research). It's a Standards Track doc, dated January 2008. The title block doesn't list any RFCs it obsoletes or updates 💅

Status of This Memo ✨

This doc specifies an Internet standards track protocol for the whole Internet community, and it's literally asking for discussion and suggestions for improvements 💬 For the current standardization state and status of this protocol, check the "Internet Official Protocol Standards" (STD 1), that's the source of truth bestie. Distribution of this memo is unlimited, so share away 🫶

Abstract 🌸

RFC 3270 defines how to support the Diffserv architecture in MPLS networks, including how to encode Diffserv Code Points (DSCPs) in an MPLS header. DSCPs may be encoded in the EXP field, while other uses of that field are not precluded. RFC 3270 makes no statement about how Explicit Congestion Notification (ECN) marking might be encoded in the MPLS header. This document defines how an operator might define some of the EXP codepoints for explicit congestion notification, without precluding other uses.

Table of Contents 🧚

1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
  1.1.  Background . . . . . . . . . . . . . . . . . . . . . . . .  3
  1.2.  Intent . . . . . . . . . . . . . . . . . . . . . . . . . .  4
  1.3.  Terminology  . . . . . . . . . . . . . . . . . . . . . . .  4
2.  Use of MPLS EXP Field for ECN  . . . . . . . . . . . . . . . .  5
3.  Per-Domain ECT Checking  . . . . . . . . . . . . . . . . . . .  7
4.  ECN-Enabled MPLS Domain  . . . . . . . . . . . . . . . . . . .  8
  4.1.  Pushing (Adding) One or More Labels to an IP Packet  . . .  8
  4.2.  Pushing One or More Labels onto an MPLS Labeled Packet . .  8
  4.3.  Congestion Experienced in an Interior MPLS Node  . . . . .  8
  4.4.  Crossing a Diffserv Domain Boundary  . . . . . . . . . . .  8
  4.5.  Popping an MPLS Label (Not the End of the Stack) . . . . .  9
  4.6.  Popping the Last MPLS Label in the Stack . . . . . . . . .  9
  4.7.  Diffserv Tunneling Models  . . . . . . . . . . . . . . . . 10
5.  ECN-Disabled MPLS Domain . . . . . . . . . . . . . . . . . . . 10
6.  The Use of More Codepoints with E-LSPs and L-LSPs  . . . . . . 10
7.  Relationship to Tunnel Behavior in RFC 3168  . . . . . . . . . 11
8.  Deployment Considerations  . . . . . . . . . . . . . . . . . . 11
  8.1.  Marking Non-ECN-Capable Packets  . . . . . . . . . . . . . 11
  8.2.  Non-ECN-Capable Routers in an MPLS Domain  . . . . . . . . 12
9.  Example Uses . . . . . . . . . . . . . . . . . . . . . . . . . 13
  9.1.  RFC 3168-Style ECN . . . . . . . . . . . . . . . . . . . . 13
  9.2.  ECN Co-Existence with Diffserv E-LSPs  . . . . . . . . . . 13
  9.3.  Congestion-Feedback-Based Traffic Engineering  . . . . . . 14
  9.4.  PCN Flow Admission Control and Flow Termination  . . . . . 14
10. Security Considerations  . . . . . . . . . . . . . . . . . . . 14
11. Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 15
Appendix A.   Extension to Pre-Congestion Notification . . . . . . 16
  A.1. Label Push onto IP Packet . . . . . . . . . . . . . . . . . 16
  A.2. Pushing Additional MPLS Labels  . . . . . . . . . . . . . . 16
  A.3. Admission Control or Flow Termination Marking Inside
       MPLS Domain . . . . . . . . . . . . . . . . . . . . . . . . 17
  A.4. Popping an MPLS Label (Not End of Stack)  . . . . . . . . . 17
  A.5. Popping the Last MPLS Label to Expose IP Header . . . . . . 17
Normative References . . . . . . . . . . . . . . . . . . . . . . . 18
Informative References . . . . . . . . . . . . . . . . . . . . . . 18

1. Introduction ✨

1.1. Background

[RFC3168] defines Explicit Congestion Notification (ECN) for IP. The main point of ECN is to let congestion get signalled without dropping packets, which is iconic behaviour honestly 🫧

[RFC3270] defines how to support the Diffserv architecture in MPLS networks, including how to encode Diffserv Code Points (DSCPs) in an MPLS header. DSCPs may be encoded in the EXP field, while other uses of that field are not precluded. RFC 3270 makes no statement about how Explicit Congestion Notification (ECN) marking might be encoded in the MPLS header 🤷‍♀️

So this document is about how an operator might define some of the EXP codepoints for explicit congestion notification, without precluding other uses. At the same time as the work adding ECN to IP [RFC3168], two proposals were made to add ECN to MPLS [Floyd][Shayman]. Those proposals fell by the wayside 😔 Now ECN for IP is a proposed standard, and there's growing interest in using pre-congestion notification (PCN) for admission control and flow termination [PCN]. That means there's a real reason to support ECN across IP networks made of MPLS-enabled domains. So it's necessary to specify the protocol for including ECN in the MPLS shim header, and the protocol behaviour of edge MPLS nodes. Main character energy for sure 💅

Quick note: in [RFC3168] there are four codepoints used for ECN marking, encoded using two bits of the IP header. The MPLS EXP field is the logical place to put ECN codepoints. But the EXP field only has 3 bits (8 codepoints), and the same field is also used to carry DSCP information. So there's a clear incentive to conserve the number of codepoints used for ECN 📦 Efficient use of the EXP field has been a focus of prior documents [Floyd] [Shayman], and this document draws on those efforts too 🙌

Also, [RFC3168] defines default usage of the ECN field, but it allows for the possibility that some Diffserv Per Hop Behaviors (PHBs) might include different specifications on how the ECN field is to be used. This document seeks to preserve that capability 🌈

1.2. Intent

Our intent is to specify how the MPLS shim header [RFC3032] should denote ECN marking, and how MPLS nodes should understand whether the transport for a packet will be ECN capable 🎯 We're offering this as a building block that different congestion-notification systems can be built from. We do not intend to specify how the resulting congestion notification is fed back to an upstream node that can mitigate congestion. For instance, unlike [Shayman], we do not specify edge-to-edge MPLS domain feedback, but we also do not preclude it.

Still, we do specify how the egress node of an MPLS domain should copy congestion notification from the MPLS shim into the encapsulated IP header, if the ECN is to be carried onward towards the IP receiver. But we do not mandate that MPLS congestion notification must be copied into the IP header for onward transmission. This document aims to be generic for any use of congestion notification in MPLS. Supporting [RFC3168] is our primary motivation. Some additional potential applications, to show how flexible our approach is, are described in Section 9. In particular, we aim to support possible future schemes that may use more than one level of congestion marking 🦋

1.3. Terminology

This document draws freely on the terminology of ECN [RFC3168] and MPLS [RFC3031]. For ease of reference we've included some definitions here, but you should refer to the references above for complete specifications of the relevant technologies 📚

  • CE: Congestion Experienced. One of the states with which a packet may be marked in a network supporting ECN. A packet is marked in this state by an ECN-capable router to indicate that this router was experiencing congestion at the time the packet arrived. 🔥
  • ECT: ECN-capable Transport. One of the ECN states that a packet may be in when it is sent by an end system. An end system marks a packet with an ECT codepoint to indicate that the endpoints of the transport protocol are ECN-capable. A router may not mark a packet as CE unless the packet was marked ECT when it arrived. 💜
  • Not-ECT: Not ECN-capable transport. An end system marks a packet with this codepoint to indicate that the endpoints of the transport protocol are not ECN-capable. A congested router cannot mark such packets as CE, so it can only drop them to indicate congestion 💀
  • EXP field. A 3-bit field in the MPLS label header [RFC3032] that may be used to convey Diffserv information (and is also used in this document to carry ECN information) 🍓
  • PHP. Penultimate Hop Popping. An MPLS operation in which the penultimate Label Switching Router (LSR) on a Label Switched Path (LSP) removes the top label from the packet before forwarding the packet to the final LSR on the LSP 🧚

Requirements Language 🫶

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

Part 1 of 5, more coming next, stay tuned 🧚💖

2. Use of MPLS EXP Field for ECN ✨

okay so the big idea is that LSRs set up for explicit congestion notification should use the EXP field in the MPLS shim header 💅 but here's the tea: [RFC3270] already uses EXP codepoints for differentiated services 😭 It doesn't rule out other compatible uses, but it clearly squeezes the room left for ECN, since the field is only 3 bits (8 codepoints) 🫶

[RFC3270] gives two ways to ask an LSR for differentiated service treatment 🦋

  • In the EXP-Inferred-PSC LSP (E-LSP) approach, different EXP codepoints in the MPLS shim header tell you the packet's per hop behavior (PHB). 💜
  • In the Label-Only-Inferred-PSC LSP (L-LSP) approach, each PHB scheduling class (PSC, as defined in [RFC3260]) gets its own MPLS label, so the LSR figures out both its forwarding and its scheduling behavior from the label alone 🌸

if an MPLS domain goes with L-LSPs, there's probably room in the EXP field for ECN codepoint(s) 🎀 with E-LSPs, though, that codepoint space is likely to be scarce 💀 this doc focuses on interworking ECN marking with the E-LSP approach, since it's the tougher problem 👑 so the same approach can also be used with L-LSPs, no cap ✨

we recommend that explicit congestion notification in MPLS should use codepoints instead of bits in the EXP field 🍓 not every PHB will need an ECN codepoint, so giving each one a dedicated bit would be wasteful 🙈 (there might also be cases where a PHB needs more than one ECN-like codepoint; see Section 9.4 for an example 🧚)

for each PHB that uses ECN marking, we assume one EXP codepoint is defined as not congestion marked (Not-CM), and at least one other codepoint is defined as congestion marked (CM) 🌈 so each ECN-using PHB eats at least two EXP codepoints, while PHBs without ECN marking only use one 🫶

we also want to use as little space in the MPLS shim header as possible to tell interior LSRs whether each packet will reach an ECN-capable transport (ECT) 🦋 but! we have to make sure an endpoint that wouldn't understand an ECN mark never gets one, otherwise it can't respond to congestion the way it should 😭 three solutions to this have been floated over time 💅

  • one option is for congested LSRs to mark the ECN field in the underlying IP header at the bottom of the label stack 🌸 many commercial LSRs already poke into the IP header for other reasons (equal cost multi-path - ECMP), but trying to find an IP header under an MPLS label stack is a whole mess 💀 the big issue is spotting that there's no IP header when non-IP packets are carried on an LSP 🙈 so we're not going to pursue this one further 🚫
  • in the scheme suggested by [Floyd], ECT and CE are squished into one bit, so 0 means ECT and 1 might mean either Not-ECT or CE 🍓 a packet already marked as congested upstream, then picked for marking at a second congested LSR, gets dropped by that second LSR, because it can't tell whether the packet was already congested or whether ECN just isn't supported by the transport 😭

this seemed kinda palatable at first, but we don't recommend it here, and here's why 💜 sometimes we want ECN marking long before actual congestion happens (e.g., pre-congestion notification) 🌈 in that case, marking rates at each LSR might be non-negligible most of the time, so the odds of a previously marked packet hitting an LSR that wants to mark it again are also non-negligible 😬 when CE and not-ECT look the same to core routers, that could lead to unacceptable drop rates 💀 if the typical marking rate at every router or LSR is p, and the typical diameter of the network of LSRs is d, then the probability that a marked packet gets chosen for marking more than once is 1-[Pr(never marked) + Pr(marked at exactly one hop)] = 1- [(1-p)^d + dp(1-p)^(d-1)] 🧮 for instance, with 6 LSRs in a row, each marking ECN with 1% probability, the chance that an already marked packet gets chosen for a second marking is 0.15% 😱 so the bit-overloading scheme would add a drop rate of 0.15% for no reason 🙅‍♀️ since most modern core networks are sized to have near-zero packet drop, dropping over one in a thousand packets unnecessarily may be unacceptable 💔

  • a third option was suggested by [Shayman] 🧚 in this scheme, interior LSRs assume the endpoints are ECN-capable, but that assumption gets checked when the final label is popped 🔍 if an interior LSR marked ECN in the EXP field of the shim header, but the IP header says the endpoints aren't ECN-capable, the edge router (or penultimate router, if penultimate hop popping is used) drops the packet 🫣 we recommend this scheme, and we call it `per-domain ECT checking', which we define more precisely in the next section ✨ its big downside is that packets can get forwarded after hitting congestion, only to be dropped at the egress of the MPLS domain 😭 the reasoning behind this choice is in Section 8.1 🎀

3. Per-Domain ECT Checking 🌸

for this discussion, the egress nodes of an MPLS domain are the nodes that pop the last MPLS label from the label stack, which exposes the IP (or, possibly, non-IP) header 🦋 note that such a node could be the ultimate or penultimate hop of an LSP, depending on whether penultimate hop popping (PHP) is used 🧚

in the per-domain ECT checking approach, the egress nodes take responsibility for checking whether the transport is ECN-capable 💅 this document doesn't specify how those nodes should pass congestion notification along, because different approaches make sense in different scenarios 🌈 but if congestion notification in the MPLS header gets copied into the IP header, the procedure MUST conform to the specification given here 👑

if congestion notification gets passed to the transport without being passed onward in the IP header first, the approach used must take similar care to check that the transport is ECN-capable before handing it ECN markings 🫶 specifically, if the transport for a particular congestion marked MPLS packet turns out not to be ECN-capable, the packet MUST be dropped at this egress node 💀

in the per-domain ECT checking approach, only the egress nodes check whether an IP packet is destined for an ECN-capable transport 🔍 so any single LSR inside an MPLS domain MUST NOT be configured to enable ECN marking unless all the egress LSRs around it are already set up to handle ECN marking 🚫 we'd better get that right, bestie 😤

we call a domain surrounded by ECN-capable egress LSRs an ECN-enabled MPLS domain 🎀 this term only means that all the egress LSRs are ECN-enabled, and some interior LSRs may not be 🦋 for instance, it's possible to use some legacy LSRs that can't support ECN in the interior of an MPLS domain, as long as all the egress LSRs are ECN-capable 🌸 and if PHP is used, the "penultimate hop" routers that do the pop operation do need to be ECN-enabled, since they're acting as egress LSRs in this context 💜

4. ECN-Enabled MPLS Domain 👑

in the following subsections, we describe various operations that can affect the ECN marking of a packet, and that may be performed at MPLS-edge and core LSRs 🧚

4.1. Pushing (Adding) One or More Labels to an IP Packet 🌈

when an IP packet gets encapsulated with an MPLS label stack, the ECN field must be translated from the IP packet into the MPLS EXP field 💅 the Not-CM (not congestion marked) state is set in the MPLS EXP field if the ECN status of the IP packet is Not-ECT, ECT(1) or ECT(0) 🫶 the CM state is set if the ECN status of the IP packet is CE 🔥 if more than one label is pushed at one time, the same value should be placed in the EXP value of all label stack entries 🍓

4.2. Pushing One or More Labels onto an MPLS Labeled Packet 🦋

the EXP field gets copied directly from the topmost label before the push over to the newly added outer label ✨ if more than one label is being pushed, the same EXP value is copied to all label-stack entries 🌸

4.3. Congestion Experienced in an Interior MPLS Node ⭐

if the EXP codepoint of the packet maps to a PHB that uses ECN marking, and the marking algorithm says the packet needs to be marked, the CM state is set (no matter whether it's already in the CM state) 💜

if the buffer is full, a packet is dropped 💀

4.4. Crossing a Diffserv Domain Boundary 🌍

if an MPLS-encapsulated packet crosses a Diffserv domain boundary, the two domains might use different encodings of the same PHB in the EXP field 🙃 in that case, the EXP field MUST be rewritten at the domain boundary 🔁 if the PHB supports ECN, then the appropriate ECN marking should also be preserved when the EXP field gets mapped at the boundary 🫶

if an MPLS-encapsulated packet in the CM state crosses from an ECN-enabled domain (as defined in Section 3) into a domain that isn't ECN-enabled, then the egress checking procedures have to run at the egress LSR of the ECN-enabled domain 😱 this means that if the encapsulated packet isn't ECN-capable, the packet MUST be dropped 💅 and this implies the egress LSR must be able to look beneath the MPLS header without popping the label stack 🔍

the related issue of Diffserv tunnel models is discussed in Section 4.7 🎀

4.5. Popping an MPLS Label (Not the End of the Stack) 🦋

when a packet has more than one MPLS label in the stack and the top label gets popped, another MPLS label is exposed 🧚 in this case, the ECN information should be transferred from the outer EXP field to the inner MPLS label like this 🌸 if the inner EXP field is Not-CM, the inner EXP field is set to the same CM or Not-CM state as the outer EXP field 💜 if the inner EXP field is CM, it stays unchanged, no matter what the outer EXP field says 🫶 note that an inner value of CM with an outer value of not-CM is anomalous, and it SHOULD be logged in some way by the LSR 🚨 bestie, log that weirdness!! 📝

4.6. Popping the Last MPLS Label in the Stack ✨

okay so when the last MPLS label gets popped, the payload is exposed 👀 If that packet is not IP and doesn't have anything equivalent to ECT, it's assumed Not-ECT and gets treated that way. So if the EXP value of the MPLS header is CM, the packet MUST be dropped 💀

Assuming an IP packet did get exposed, we have to check whether that packet is ECT. A Not-ECT packet MUST be dropped if the EXP field is CM 🙅‍♀️

For the rest of this section, we're describing the behaviour that's required if the ECN info is going to be carried from the MPLS header into the exposed IP header for onward transmission. As noted in Section 1.2, this isn't mandated by this document, but an operator may select it 🎀

If the inner IP packet is Not-ECT, its ECN field stays unchanged when the EXP field is Not-CM. If the ECN field of the inner packet is set to ECT(0), ECT(1), or CE, the ECN field stays unchanged when the EXP field is set to Not-CM. The ECN field gets set to CE if the EXP field is CM ✨ Heads up: an inner CE with an outer Not-CM should be considered anomalous, and the LSR SHOULD log it in some way 🔥

4.7. Diffserv Tunneling Models 🦋

[RFC3270] describes three tunneling models for Diffserv support across MPLS Domains, called the uniform, short pipe, and pipe models. The big difference is whether the Diffserv treatment a packet gets while it travels along a particular LSP is carried to the ingress of the last hop, to the egress of the last hop, or beyond the last hop. Depending on which mode an operator prefers, the EXP value or DSCP value of an exposed header after a label pop may or may not depend on the EXP value of the label that got removed. We believe that for ECN marking, these models should only apply to how the Diffserv PHB is encoded in the EXP value. The choice of codepoint for ECN should always be made using the procedures described above, no matter which tunneling model is in use 💅

5. ECN-Disabled MPLS Domain 🌸

If ECN is not enabled on all the egress LSRs of a domain, ECN MUST NOT be enabled on any LSRs throughout the domain. If congestion is experienced on any LSR in an ECN-disabled MPLS domain, packets MUST be dropped; they MUST NOT be marked. The exact algorithm for deciding when to drop packets during congestion (e.g., tail-drop, RED, etc.) is a local matter for the operator of the domain 🧚

6. The Use of More Codepoints with E-LSPs and L-LSPs 🍓

[RFC3270] gives different options with E-LSPs and L-LSPs, and some of those could potentially provide ample EXP codepoints for ECN. Real talk though, deploying L-LSPs vs. E-LSPs has a lot of implications, like platform support and operational complexity 😭 The above ECN MPLS solution should give some flexibility. If the operator has deployed one L-LSP per PHB scheduling class, then EXP space is a non-issue, and it could be used to get more sophisticated ECN behaviour if it's required. If the operator wants to stick with E-LSPs and uses a handful of EXP codepoints for Diffserv, it may be desirable to operate with a minimum number of extra ECN codepoints, even if that comes with some compromise on ECN optimality. See Section 9 for discussion of some possible deployment scenarios 👑

We note that in a network where L-LSPs are used, ECN marking SHOULD NOT cause packets from the same microflow, but with different ECN markings, to be sent on different LSPs. As discussed in [RFC3270], packets of a single microflow should always travel on the same LSP to avoid possible misordering. So ECN marking of packets on L-LSPs SHOULD only affect the EXP value of the packets 🫶

7. Relationship to Tunnel Behavior in RFC 3168 🌈

[RFC3168] defines two modes for encapsulating ECN-marked IP packets inside additional IP headers when tunnels are used. The two modes are the "full functionality" and "limited functionality" modes. In full functionality mode, the ECT info from the inner header is copied to the outer header at the tunnel ingress, but the CE info is not. In limited functionality mode, neither ECT nor CE info is copied to the outer header, so ECN can't be applied to the encapsulated packet 😤

The behaviour in Section 4 of this document kind of resembles the "full functionality" mode, since it conveys some info from inner to outer header, and since it enables full ECN support along the MPLS LSP (which is basically like an IP tunnel in this context) ✨ But it's different in one way: the CE info is conveyed from the inner header to the outer header. The original reason for this different design choice was to give interior routers and LSRs more info about upstream marking in multi-bottleneck cases. For instance, the flow termination marking mechanism proposed for PCN only considers packets for marking that haven't already been marked upstream. Unless the existing flow termination marking is copied from the inner to the outer header at tunnel ingress, the mechanism doesn't terminate enough traffic when anomalous events hit multiple domains at once 🚨 [RFC3168] doesn't give any reasons against conveying CE info from the inner header to the outer one in the "full functionality" mode. Plus, [RFC4301] specifies that the ECN marking should be copied from the inner header to the outer header in IPSEC tunnels, which lines up with the approach defined here. [BRISCOE-ECN] talks about this issue in more detail. Bottom line: the approach in Section 4 appears to be both a sound technical choice and consistent with the current state of thinking in the IETF 🔥

8. Deployment Considerations 🎀

8.1. Marking Non-ECN-Capable Packets 🙈

What are the consequences of marking a packet that is not ECN-capable? Even if it will be dropped before leaving the domain, doesn't this consume resources unnecessarily?

The problem only arises if there's congestion downstream of an earlier congested queue in the same MPLS domain. Congested LSRs downstream might forward packets that are already marked, even though they'll get dropped later when the inner IP header is found to be Not-ECT on decapsulation. Those packets might cause the downstream LSRs to mark (or drop) other packets that they otherwise wouldn't have had to 😬

We expect congestion will typically be rare in MPLS networks, but it might not be. The extra unnecessary load at downstream LSRs won't be more than the fraction of marked packets coming from upstream LSRs, even in the worst case where no transports are ECN-capable. So the amount of unnecessary marking (or drop) on an LSR won't be more than the product of its local marking rate and the marking rate due to upstream LSRs within the same domain. That's typically the product of two small (often zero) probabilities 🧮

This is why we went with the per-domain ECT checking approach. The most likely effect is a very slightly increased marking rate, which would result in very slightly higher drop only for non-ECN-capable transports. We chose not to use the [Floyd] alternative, which introduced a low but persistent level of unnecessary packet drop for all time, even for ECN-capable transports. Although that scheme didn't carry traffic to the edge of the MPLS domain only to be dropped on decapsulation, we felt our minor inefficiency was a small price to pay, and it would get smaller still if ECN deployment widened 🌱

A partial solution would be to preferentially drop packets arriving at a congested router that were already marked. There's no solution to the problem of marking a packet when congestion is caused by another packet that should have been dropped. However, the chance of that happening is very low, and the consequences aren't significant. It just causes an application to very occasionally slow down its rate when it didn't have to 🐢

8.2. Non-ECN-Capable Routers in an MPLS Domain 🧱

What if an MPLS domain wants to use ECN, but not all legacy routers are able to support it?

If the legacy router(s) are used in the interior, this isn't a problem. They'll simply have to drop the packets if they're congested, rather than mark them, which is the standard behaviour for IP routers that aren't ECN-enabled 👍

If a legacy router were used as an egress router, it wouldn't be able to check the ECN-capability of the transport correctly. An operator in this position wouldn't be able to use this solution, and therefore MUST NOT enable ECN unless all egress routers are ECN-capable 🚫

9. Example Uses 💖

9.1. RFC 3168-Style ECN 💜

[RFC3168] proposes the use of ECN in TCP, and it introduces the use of ECN-Echo and Congestion Window Reduced (CWR) flags in the TCP header for initialization. The TCP sender responds accordingly (such as by not increasing the congestion window) when it receives an ECN-Echo (ECE) ACK packet, that is, an ACK packet with the ECN-Echo flag set in the TCP header. Then the sender knows that congestion was encountered in the network on the path from the sender to the receiver 🥹

It would be possible to enable ECN in an MPLS domain for Diffserv PHBs like AF and best efforts that are expected to be used by TCP and similar transports (e.g., DCCP [RFC4340]). Then, end-to-end congestion control in transports that understand ECN would be able to respond to approaching congestion on LSRs without having to rely on packet discard to signal congestion. Honestly, that's so clean 🦋

9.2. ECN Co-Existence with Diffserv E-LSPs 🌷

Many operators today have deployed Diffserv using the E-LSP approach of [RFC3270]. In many cases, the number of PHBs used is less than 8, so there are still codepoints available in the EXP space. If an operator wants to support ECN for a single PHB, they can do that by allocating a second codepoint to the PHB for the CM state of that PHB and keeping the old codepoint for the Not-CM state. An operator with only four deployed PHBs could, of course, enable ECN marking on all of those PHBs. It's easy to imagine cases where some PHBs benefit more from ECN than others. For example, an operator might use ECN on a premium data service but not on a PHB used for best-effort Internet traffic 💅

As an illustrative example of how the EXP field might be used in this case, consider an operator who is using the aggregated service classes proposed in [TSVWG]. He may choose to support ECN only for the Assured Elastic Treatment Aggregate, using EXP codepoint 010 for the Not-CM state and 011 for the CM state. All other codepoints could be the same as in [TSVWG]. Of course, any other combination of EXP values can be used, depending on the specific set of PHBs and marking conventions used within that operator's network 🌈

9.3. Congestion-Feedback-Based Traffic Engineering ✨

okay so Shayman's traffic engineering [Shayman] is another awesome example of ECN feedback working inside an MPLS domain 💅 The idea is that an egress LSR tells an ingress LSR about congestion, so the network can fix it with dynamic traffic engineering, like moving flows onto an alternate path 🦋 Shayman proposed a new Resource Reservation Protocol (RSVP) message for this. The egress LSR sends it to the ingress LSR, and transit LSRs just ignore it 🙈 It's basically a way to say "hey, there's congestion along this path" 🫶

Here's the tea: this isn't the same style of congestion notification that endpoints get under [RFC3168]. [Shayman] keeps its scope to the MPLS domain only 💜 Even so, this kind of ECN use in an MPLS domain could totally make use of the ECN encoding in the MPLS header that is defined in this document 🌸

9.4. PCN Flow Admission Control and Flow Termination 🎀

[PCN] proposes using pre-congestion notification (PCN) on routers inside an edge-to-edge Diffserv region 🌈 The goal is to control admission of new flows into the region and, if needed, terminate existing flows when there are disasters or other weird routing events 💀 The vibe is that the current level of PCN marking gets picked up by the signaling used to start each flow, so the admission control decision for the whole region gets made at once 👑 For example, extensions to RSVP [LEFAUCHEUR] and Next Steps in Signaling (NSIS) [NSIS], [ARUMAITHURAI] have been proposed 🧚

If LSRs can mark packets to signify congestion in MPLS, then PCN marking could be used for admission control and flow termination across a Diffserv region 🔥 This works whether the region has pure IP routers, MPLS LSRs, or both, which is so iconic 😭 Honestly, the solution could be a bit more efficient to implement if aggregates could identify themselves by their MPLS label 💅 Appendix A describes the mechanisms by which the necessary markings for PCN could be carried in the MPLS header 🍓

10. Security Considerations ✨

We believe no new vulnerabilities are introduced by this document 🛡️

We've thought about whether malicious sources could take advantage of the fact that interior LSRs will mark packets that are Not-ECT, while relying on their egress LSR to drop them 👀 This might let sources engineer a situation where more traffic is carried across an MPLS domain than should be 💀 But even if we hadn't added this feature, those sources could still stop these LSRs from dropping this traffic, just by setting ECT in the first place 🙄 So it's not really a new problem, periodt 💅

An ECN sender can use the ECN nonce [RFC3540] to detect a misbehaving receiver 🦋 The ECN nonce works correctly across an MPLS domain without needing any specific support from the proposal in this document ✨ The nonce doesn't need to be in the MPLS shim header to detect a misbehaving receiver 💖 As long as the nonce is in the IP header when the ECN information is copied from the last MPLS shim header, it will be overwritten if an LSR has experienced congestion 🔥 That's all the sender needs to detect a misbehaving receiver 👑

Now, if there were a need for an ECN nonce in the MPLS shim header (for example, to detect if one LSR were erasing the markings of an upstream LSR in the same domain), we believe this proposal does not preclude adding an ECN nonce capability later for specific DSCPs 🌈 It also doesn't preclude any other use of the EXP codepoints, so there's plenty of room to grow 🧚

Security is not a vibe to skip. Read this section like your whole network depends on it, because the marking and dropping decisions really do matter 🚨

11. Acknowledgments 🫶

Thanks to K.K. Ramakrishnan and Sally Floyd for getting us thinking about this in the first place 💜 They also gave advice on tunneling of ECN packets 🎀 Thanks as well to Sally Floyd, Joe Babiarz, Ben Niven-Jenkins, Phil Eardley, Ruediger Geib, and Magnus Westerlund for their comments on the document 🌸 You all are literally the best 😭

Appendix A. Extension to Pre-Congestion Notification 🌈

This appendix describes how the mechanisms in the body of the document can be extended to support PCN [PCN] ✨ The point here is to show that the mechanisms are easily extended to more complex scenarios than ECN, especially when more codepoints are needed 💅 You can safely ignore this appendix if you're only supporting ECN 🙈 Note that the PCN standards are still very much under development at the time of writing 🚧 So the precise details in this appendix may change, and this appendix is for illustrative purposes only, bestie 🍓

The relevant aspects of PCN for this discussion are:

  • PCN uses 3 states rather than 2 for ECN 🦋 These are called admission marked (AM), termination marked (TM), and not marked (NM) states. (See Section 9.4 for further discussion of PCN and the possibility of using fewer codepoints.)
  • A packet can go from NM to AM, from NM to TM, or from AM to TM, but no other transition is possible ✨
  • The determination of whether a packet is subject to PCN is based on the PHB of the packet 🔥

So, to fully support PCN in an MPLS domain for a particular PHB, a total of 3 codepoints need to be allocated for that PHB 💖 Those 3 codepoints represent the admission marked (AM), termination marked (TM), and not marked (NM) states 👑 The procedures described in Section 4 above need a slight tweak to support this scenario 🎀 The following procedures are invoked when the topmost DSCP or EXP value indicates a PHB that supports PCN 🧚

A.1. Label Push onto IP Packet 💅

If the IP packet header indicates AM, set the EXP value of all entries in the label stack to AM 🌸 If the IP packet header indicates TM, set the EXP value of all entries in the label stack to TM 🦋 For any other marking of the IP header, set the EXP value of all entries in the label stack to NM 🫶

A.2. Pushing Additional MPLS Labels ✨

The procedures of Section 4.2 apply 💜

A.3. Admission Control or Flow Termination Marking Inside MPLS Domain 🔥

The EXP value can be set to AM or TM according to the same procedures as described in [BRISCOE-CL] 🌈 For the purposes of this document, it doesn't matter exactly which algorithms are used to decide when to set AM or TM 🙌 All that matters is that if a router would have marked AM (or TM) in the IP header, it sets the EXP value in the MPLS header to the AM (or TM) codepoint 💖

A.4. Popping an MPLS Label (Not End of Stack) 🎀

When popping an MPLS Label exposes another MPLS label, the AM or TM marking should be transferred to the exposed EXP field in the following manner:

  • If the inner EXP value is NM, then it should be set to the same marking state as the EXP value of the popped label stack entry 🍓
  • If the inner EXP value is AM, it should be unchanged if the popped EXP value was AM, and it should be set to TM if the popped EXP value was TM 🦋 If the popped EXP value was NM, this should be logged in some way, and the inner EXP value should be unchanged 📝
  • If the inner EXP value is TM, it should be unchanged whatever the popped EXP value was, but any EXP value other than TM should be logged 💅

A.5. Popping the Last MPLS Label to Expose IP Header 👑

When popping the last MPLS Label exposes the IP header, there are two cases to consider:

  • the popping LSR is not the egress router of the PCN region, in which case AM or TM marking should be transferred to the exposed IP header field 🌸; or
  • the popping LSR is the egress router of the PCN region 🧚

In the latter case, the behavior of the egress LSR is defined in [PCN] and is beyond the scope of this document 🙈 In the former case, the marking should be transferred from the popped MPLS header to the exposed IP header as follows:

  • If the inner IP header value is neither AM nor TM, and the EXP value was NM, then the IP header should be unchanged 💜 For any other EXP value, the IP header should be set to the same marking state as the EXP value of the popped label stack entry ✨
  • If the inner IP header value is AM, it should be unchanged if the popped EXP value was AM, and it should be set to TM if the popped EXP value was TM 🦋 If the popped EXP value was NM, this should be logged in some way and the inner IP header value should be unchanged 📝
  • If the IP header value is TM, it should be unchanged whatever the popped EXP value was, but any EXP value other than TM should be logged 🔥

Normative References 📚

  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 🔑
  • [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, January 2001. 🏷️
  • [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack Encoding", RFC 3032, January 2001. 🧩
  • [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, September 2001. 🌈
  • [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-Protocol Label Switching (MPLS) Support of Differentiated Services", RFC 3270, May 2002. 💖
  • [RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005. 🔐

Informative References 🧠

  • [ARUMAITHURAI] Arumaithurai, M., "NSIS PCN-QoSM: A Quality of Service Model for Pre-Congestion Notification (PCN)", Work in Progress, September 2007. 🚧
  • [BRISCOE-CL] Briscoe, B., "Pre-Congestion Notification Marking", Work in Progress, October 2006. 🚧
  • [BRISCOE-ECN] Briscoe, B., "Layered Encapsulation of Congestion Notification", Work in Progress, July 2007. 🚧
  • [Floyd] Ramakrishnan, K., Floyd, S., and B. Davie, "A Proposal to Incorporate ECN in MPLS", Work in Progress, June 1999. 🚧
  • [LEFAUCHEUR] Faucheur, F., Charny, A., Briscoe, B., Eardley, P., Barbiaz, J., and K. Chan, "RSVP Extensions for Admission Control over Diffserv using Pre-congestion Notification (PCN)", Work in Progress, June 2006. 🚧
  • [NSIS] Bader, A., Westberg, L., Karagiannis, G., Cornelia, C., and T. Phelan, "RMD-QOSM - The Resource Management in Diffserv QOS Model", Work in Progress, November 2007. 🚧
  • [PCN] Eardley, P., "Pre-Congestion Notification Architecture", Work in Progress, November 2007. 🚧
  • [RFC3260] Grossman, D., "New Terminology and Clarifications for Diffserv", RFC 3260, April 2002. 📖
  • [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit Congestion Notification (ECN) Signaling with Nonces", RFC 3540, June 2003. 📖
  • [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, March 2006. 📖
  • [Shayman] Shayman, M. and R. Jaeger, "Using ECN to Signal Congestion Within an MPLS Domain", Work in Progress, November 2000. 🚧
  • [TSVWG] Chan, K., Babiarz, J., and F. Baker, "Aggregation of DiffServ Service Classes", Work in Progress, November 2007. 🚧

Authors' Addresses 💌

The squad behind this one 🦋 Here's where to find all three of them:

Bruce Davie
Cisco Systems, Inc.
1414 Mass. Ave.
Boxborough, MA  01719
USA

EMail: bsd@cisco.com
Bob Briscoe
BT Research
B54/77, Sirius House
Adastral Park
Martlesham Heath
Ipswich
Suffolk  IP5 3RE
United Kingdom

EMail: bob.briscoe@bt.com
June Tay
BT Research
B54/77, Sirius House
Adastral Park
Martlesham Heath
Ipswich
Suffolk  IP5 3RE
United Kingdom

EMail: june.tay@bt.com

Okay, copyright time, and it's giving serious legal energy 💅 The IETF Trust owns the copyright on this document, dated 2008. ✨

This document is subject to the rights, licenses and restrictions contained in BCP 78. Apart from what BCP 78 says, the authors keep all their rights. 🌸

This document and the information in it are provided on an "AS IS" basis, and the contributor, the organization they represent or are sponsored by (if any), the Internet Society, the IETF Trust, and the Internet Engineering Task Force disclaim all warranties. That covers both express and implied warranties, including, but not limited to, any warranty that using the information herein won't infringe any rights, and any implied warranties of merchantability or fitness for a particular purpose. 💀

THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🔥

Let's talk IP, bestie. The IETF doesn't take a side on whether any intellectual property rights claimed over the tech in this doc are valid or how far they reach. It also doesn't say whether licenses for those rights are available. And it hasn't tried to dig up any such rights itself. 🧚 If you want the rules on rights in RFC documents, check out BCP 78 and BCP 79. ✨

Copies of IPR disclosures sent to the IETF Secretariat, plus any assurances of licenses that are going to be available, can be found in the IETF's online IPR repository at http://www.ietf.org/ipr. The same goes for the results of any attempt to get a general license or permission to use those proprietary rights, whether that's for implementers or users of this spec. 🎀

The IETF also invites anyone to flag any copyrights, patents, patent applications, or other proprietary rights that might cover technology needed to implement this standard. Send that info to the IETF at ietf-ipr@ietf.org. Speak up, queen! 👑🫶

📦 where this page comes from