RFC 6040

Tunnelling of Explicit Congestion Notification

📅 published November 2010, from ietf

Internet Engineering Task Force (IETF) 🎀 Request for Comments: 6040 💖 Author: B. Briscoe, from BT ✨ Date: November 2010 🌸 Category: Standards Track 👑 ISSN: 2070-1721 It updates RFC 3168, RFC 4301 and RFC 4774 🫶

Tunnelling of Explicit Congestion Notification 🦋

Abstract ✨

okay so this doc redefines how the explicit congestion notification (ECN) field of the IP header should be built when a packet enters and leaves any IP-in-IP tunnel 🚇 On the way in (encapsulation), it updates RFC 3168 so all IP-in-IP tunnels (v4 or v6) match RFC 4301 IPsec ECN processing 💅 On the way out (decapsulation), it updates both RFC 3168 and RFC 4301 to add new behaviours for combinations of inner and outer headers that nobody was using before 🙈 The new rules make sure the ECN field gets propagated correctly across a tunnel whether it's signalling one or two severity levels of congestion. Before this, only one severity level was supported 💀 Tunnel endpoints can be updated in any order without messing up existing uses of the ECN field, so backward compatibility is totally preserved 🧚 Still, operators who want two severity levels (e.g., for pre-congestion notification -- PCN) can require compliance with this new spec 🔥 There's also a thorough analysis of the reasoning behind these changes and what they imply 📚 In the unlikely event the new rules don't meet a specific need, RFC 4774 gives guidance on designing alternate ECN semantics, and this document extends that guidance to cover tunnelling issues 🌈

Status of This Memo ⭐

This is an Internet Standards Track document. 🎀

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 5741. 💜

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc6040. 🍓

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. 🦋

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License. 💅

Table of Contents 📜

   1. Introduction ....................................................4
      1.1. Scope ......................................................5
   2. Terminology .....................................................6
   3. Summary of Pre-Existing RFCs ....................................7
      3.1. Encapsulation at Tunnel Ingress ............................7
      3.2. Decapsulation at Tunnel Egress .............................8
   4. New ECN Tunnelling Rules ........................................9
      4.1. Default Tunnel Ingress Behaviour ..........................10
      4.2. Default Tunnel Egress Behaviour ...........................10
      4.3. Encapsulation Modes .......................................12
      4.4. Single Mode of Decapsulation ..............................14
   5. Updates to Earlier RFCs ........................................15
      5.1. Changes to RFC 4301 ECN Processing ........................15
      5.2. Changes to RFC 3168 ECN Processing ........................16
      5.3. Motivation for Changes ....................................17
           5.3.1. Motivation for Changing Encapsulation ..............17
           5.3.2. Motivation for Changing Decapsulation ..............18
   6. Backward Compatibility .........................................21
      6.1. Non-Issues Updating Decapsulation .........................21
      6.2. Non-Update of RFC 4301 IPsec Encapsulation ................21
      6.3. Update to RFC 3168 Encapsulation ..........................22
   7. Design Principles for Alternate ECN Tunnelling Semantics .......22
   8. Security Considerations ........................................24
   9. Conclusions ....................................................26
   10. Acknowledgements ..............................................26
   11. References ....................................................27
      11.1. Normative References .....................................27
      11.2. Informative References ...................................27
   Appendix A.  Early ECN Tunnelling RFCs ............................29
   Appendix B.  Design Constraints ...................................29
     B.1.  Security Constraints ......................................29
     B.2.  Control Constraints .......................................31
     B.3.  Management Constraints ....................................32
   Appendix C.  Contribution to Congestion across a Tunnel ...........33
   Appendix D.  Compromise on Decap with ECT(1) Inner and ECT(0)
                Outer ................................................34
   Appendix E.  Open Issues ..........................................35

1. Introduction 🦋

okay so ECN ([RFC3168]) lets a forwarding element (e.g., a router) signal that congestion is starting without having to drop packets 🙌 Instead, it can explicitly mark a proportion of packets in the two-bit ECN field in the IP header (Table 1 recaps the ECN codepoints) ✨

The outer header of an IP packet can encapsulate one or more IP headers for tunnelling 🚇 A forwarding element using ECN to signify congestion will only mark the immediately visible outer IP header, bestie. Then, when a tunnel decapsulator removes that outer header, it follows rules to propagate the congestion markings by combining the ECN fields of the inner and outer IP headers into one outgoing IP header 🫶

This document updates those rules for IPsec [RFC4301] and non-IPsec [RFC3168] tunnels, adding new behaviours for combinations of inner and outer headers that were previously unused 💖 It also updates the ingress behaviour of RFC 3168 tunnels so it matches RFC 4301 tunnels 💅 Tunnel endpoints that follow the updated rules stay backward compatible when they interwork with endpoints following RFC 4301, RFC 3168, or any earlier specification 🌸

When RFC 3168 defined ECN and its tunnelling, only the minimum necessary changes to the ECN field were propagated through tunnel endpoints, just enough for the basic ECN mechanism to work 🙈 The reason was a worry that the ECN field might be toggled to communicate between a secure site and someone on the public Internet, which is a covert channel 🕵️‍♀️ That's because a mutable field like ECN cannot be protected by IPsec's integrity mechanisms, since it has to be able to change as it traverses the Internet 😭

Nonetheless, the latest IPsec architecture [RFC4301] considered a bandwidth limit of two bits per packet on a covert channel to be a manageable risk 💅 So, for simplicity, an RFC 4301 ingress copied the whole ECN field to encapsulate a packet. RFC 4301 also dispensed with the two modes of RFC 3168, one of which partially copied the ECN field and the other of which blocked all propagation of ECN changes 🧚

Unfortunately, this totally reasonable sequence of standards actions resulted in a perverse outcome 💀 Non-IPsec tunnels (RFC 3168) blocked the two-bit covert channel, while IPsec tunnels (RFC 4301) did not, at least not at the ingress. At the egress, both IPsec and non-IPsec tunnels still partially restricted propagation of the full ECN field 🙃

The trigger for the changes in this document was the introduction of pre-congestion notification (PCN [RFC5670]) to the IETF Standards Track 🔥 PCN needs the ECN field to be copied at a tunnel ingress, and it needs four states of congestion signalling to be propagated at the egress. But pre-existing tunnels only propagate three states in the ECN field 😤

This document draws on currently unused (CU) combinations of inner and outer headers to add tunnelling of four-state congestion signalling to RFC 3168 and RFC 4301 🌈 Operators of tunnels who specifically want four states can require that all their tunnels comply with this specification. However, this is not a fork in the RFC series 🙅‍♀️ It's an update that can be deployed first by those who need it, and then by all tunnel endpoint implementations (RFC 4301, RFC 3168, RFC 2481, RFC 2401, RFC 2003), which can safely be updated to this new specification as part of general code maintenance 🛠️ This will gradually add support for four congestion states to the Internet, and existing three-state schemes will keep working as before 🦋

In fact, this document is the opposite of a fork 👑 At the same time as supporting a fourth state, the opportunity was taken to draw together divergent ECN tunnelling specifications into a single consistent behaviour, harmonising differences such as the perverse covert channel treatment. Then any tunnel can be deployed unilaterally, and it will support the full range of congestion control and management schemes without any modes or configuration 🎀 Further, any host or router can expect the ECN field to behave the same way, whatever type of tunnel might intervene in the path 💜

1.1. Scope 🌸

This document only concerns wire protocol processing of the ECN field at tunnel endpoints. It makes no changes or recommendations concerning algorithms for congestion marking or congestion response 🙌

This document specifies common ECN field processing at encapsulation and decapsulation for any IP-in-IP tunnelling, whether IPsec or non-IPsec tunnels. It applies irrespective of whether IPv4 or IPv6 is used for either the inner or outer headers. It applies for packets with any destination address type, whether unicast or multicast. It applies as the default for all Diffserv per-hop behaviours (PHBs), unless stated otherwise in the specification of a PHB (but Section 4 strongly deprecates such exceptions). It's meant to be a good trade-off between somewhat conflicting security, control, and management requirements 🧚

[RFC2983] is a comprehensive primer on differentiated services and tunnels 📚 Because ECN raises similar issues to differentiated services when it interacts with tunnels, useful concepts introduced in RFC 2983 are used throughout, with brief recaps of the explanations where necessary 🫶

2. Terminology ✨

Okay bestie, vocab time 💅 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

Table 1 recaps the names of the ECN codepoints [RFC3168] 🌸

  +------------------+----------------+---------------------------+
  | Binary codepoint | Codepoint name | Meaning                   |
  +------------------+----------------+---------------------------+
  |        00        | Not-ECT        | Not ECN-capable transport |
  |        01        | ECT(1)         | ECN-capable transport     |
  |        10        | ECT(0)         | ECN-capable transport     |
  |        11        | CE             | Congestion experienced    |
  +------------------+----------------+---------------------------+

       Table 1: Recap of Codepoints of the ECN Field [RFC3168]
                          in the IP Header

Further terminology used within this document, and it's giving precision 🧚

Encapsulator: The tunnel endpoint function that adds an outer IP header to tunnel a packet (also termed the 'ingress tunnel endpoint' or just the 'ingress' where the context is clear).

Decapsulator: The tunnel endpoint function that removes an outer IP header from a tunnelled packet (also termed the 'egress tunnel endpoint' or just the 'egress' where the context is clear).

Incoming header: The header of an arriving packet before encapsulation.

Outer header: The header added to encapsulate a tunnelled packet.

Inner header: The header encapsulated by the outer header.

Outgoing header: The header constructed by the decapsulator using logic that combines the fields in the outer and inner headers.

Copying ECN: On encapsulation, setting the ECN field of the new outer header to be a copy of the ECN field in the incoming header.

Zeroing ECN: On encapsulation, clearing the ECN field of the new outer header to Not-ECT ("00").

Resetting ECN: On encapsulation, setting the ECN field of the new outer header to be a copy of the ECN field in the incoming header except the outer ECN field is set to the ECT(0) codepoint if the incoming ECN field is CE. 💖

3. Summary of Pre-Existing RFCs 🎀

This section is informative not normative, as it recaps pre-existing RFCs. Earlier relevant RFCs that were either Experimental or incomplete with respect to ECN tunnelling (RFC 2481, RFC 2401, and RFC 2003) are briefly outlined in Appendix A. The question of whether tunnel implementations used in the Internet comply with any of these RFCs is not discussed.

3.1. Encapsulation at Tunnel Ingress

At the encapsulator, the controversy has been over whether to propagate information about congestion experienced on the path so far into the outer header of the tunnel. Drama 😭

Specifically, RFC 3168 says that, if a tunnel fully supports ECN (termed a 'full-functionality' ECN tunnel in [RFC3168]), the encapsulator must not copy a CE marking from the incoming header into the outer header that it creates. Instead, the encapsulator must set the outer header to ECT(0) if the ECN field is marked CE in the arriving IP header. We term this 'resetting' a CE codepoint.

However, the new IPsec architecture in [RFC4301] reverses this rule, stating that the encapsulator must simply copy the ECN field from the incoming header to the outer header. Plot twist 🙈

RFC 3168 also provided a Limited Functionality mode that turns off ECN processing over the scope of the tunnel by setting the outer header to Not-ECT ("00"). Then, such packets will be dropped to indicate congestion, rather than marked with ECN. This is necessary for the ingress to interwork with legacy decapsulators ([RFC2481], [RFC2401], and [RFC2003]) that do not propagate ECN markings added to the outer header. Otherwise, such legacy decapsulators would throw away congestion notifications before they reached the transport layer. Not okay 💀

Neither Limited Functionality mode nor Full Functionality mode are used by an RFC 4301 IPsec encapsulator, which simply copies the incoming ECN field into the outer header. An earlier key-exchange phase ensures an RFC 4301 ingress will not have to interwork with a legacy egress that does not support ECN. ✨

These pre-existing behaviours are summarised in Figure 1 🦋

  +-----------------+-----------------------------------------------+
  | Incoming Header |             Departing Outer Header            |
  | (also equal to  +---------------+---------------+---------------+
  | departing Inner |  RFC 3168 ECN |  RFC 3168 ECN | RFC 4301 IPsec|
  |     Header)     |    Limited    |     Full      |               |
  |                 | Functionality | Functionality |               |
  +-----------------+---------------+---------------+---------------+
  |    Not-ECT      |   Not-ECT     |   Not-ECT     |   Not-ECT     |
  |     ECT(0)      |   Not-ECT     |    ECT(0)     |    ECT(0)     |
  |     ECT(1)      |   Not-ECT     |    ECT(1)     |    ECT(1)     |
  |       CE        |   Not-ECT     |    ECT(0)     |      CE       |
  +-----------------+---------------+---------------+---------------+


  Figure 1: IP-in-IP Encapsulation: Recap of Pre-Existing Behaviours

3.2. Decapsulation at Tunnel Egress

RFC 3168 and RFC 4301 specify the decapsulation behaviour summarised in Figure 2. The ECN field in the outgoing header is set to the codepoint at the intersection of the appropriate arriving inner header (row) and arriving outer header (column). 🍓

           +---------+------------------------------------------------+
           |Arriving |            Arriving Outer Header               |
           |   Inner +---------+------------+------------+------------+
           |  Header | Not-ECT | ECT(0)     | ECT(1)     |     CE     |
           +---------+---------+------------+------------+------------+
 RFC 3168->| Not-ECT | Not-ECT |Not-ECT     |Not-ECT     |  <drop>    |
 RFC 4301->| Not-ECT | Not-ECT |Not-ECT     |Not-ECT     |Not-ECT     |
           |  ECT(0) |  ECT(0) | ECT(0)     | ECT(0)     |     CE     |
           |  ECT(1) |  ECT(1) | ECT(1)     | ECT(1)     |     CE     |
           |    CE   |      CE |     CE     |     CE     |     CE     |
           +---------+---------+------------+------------+------------+

In pre-existing RFCs, the ECN field in the outgoing header was set to the codepoint at the intersection of the appropriate arriving inner header (row) and arriving outer header (column), or the packet was dropped where indicated.

    Figure 2: IP in IP Decapsulation; Recap of Pre-Existing Behaviour

The behaviour in the table derives from the logic given in RFC 3168 and RFC 4301, briefly recapped as follows:

  • On decapsulation, if the inner ECN field is Not-ECT the outer is ignored. RFC 3168 (but not RFC 4301) also specified that the decapsulator must drop a packet with a Not-ECT inner and CE in the outer.
  • In all other cases, if the outer is CE, the outgoing ECN field is set to CE; otherwise, the outer is ignored and the inner is used for the outgoing ECN field.

Section 9.2.2 of RFC 3168 also made it an auditable event for an IPsec tunnel "if the ECN Field is changed inappropriately within an IPsec tunnel...". Inappropriate changes were not specifically enumerated. RFC 4301 did not mention inappropriate ECN changes. 🙃

4. New ECN Tunnelling Rules 👑

The standards actions below in Section 4.1 (ingress encapsulation) and Section 4.2 (egress decapsulation) define new default ECN tunnel processing rules for any IP packet (v4 or v6) with any Diffserv codepoint. Main character energy ✨

If these defaults do not meet a particular requirement, an alternate ECN tunnelling scheme can be introduced as part of the definition of an alternate congestion marking scheme used by a specific Diffserv PHB (see [RFC4774] and Section 5 of [RFC3168]). When designing such alternate ECN tunnelling schemes, the principles in Section 7 should be followed. However, alternate ECN tunnelling schemes SHOULD be avoided whenever possible as the deployment burden of handling exceptional PHBs in implementations of all affected tunnels should not be underestimated. There is no requirement for a PHB definition to state anything about ECN tunnelling behaviour if the default behaviour in the present specification is sufficient. 🌈

4.1. Default Tunnel Ingress Behaviour

Two modes of encapsulation are defined here; a REQUIRED 'normal mode' and a 'compatibility mode', which is for backward compatibility with tunnel decapsulators that do not understand ECN. Note that these are modes of the ingress tunnel endpoint only, not the whole tunnel. Section 4.3 explains why two modes are necessary and specifies the circumstances in which it is sufficient to solely implement normal mode. 🫶

Whatever the mode, an encapsulator forwards the inner header without changing the ECN field.

In normal mode, an encapsulator compliant with this specification MUST construct the outer encapsulating IP header by copying the two-bit ECN field of the incoming IP header. In compatibility mode, it clears the ECN field in the outer header to the Not-ECT codepoint (the IPv4 header checksum also changes whenever the ECN field is changed). These rules are tabulated for convenience in Figure 3. 🔥

           +-----------------+-------------------------------+
           | Incoming Header |    Departing Outer Header     |
           | (also equal to  +---------------+---------------+
           | departing Inner | Compatibility |    Normal     |
           |     Header)     |     Mode      |     Mode      |
           +-----------------+---------------+---------------+
           |    Not-ECT      |   Not-ECT     |   Not-ECT     |
           |     ECT(0)      |   Not-ECT     |    ECT(0)     |
           |     ECT(1)      |   Not-ECT     |    ECT(1)     |
           |       CE        |   Not-ECT     |      CE       |
           +-----------------+---------------+---------------+

             Figure 3: New IP in IP Encapsulation Behaviours

4.2. Default Tunnel Egress Behaviour ✨

Okay bestie, so when the packet reaches the tunnel egress and it's time to unwrap the inner header, a compliant tunnel egress MUST set the outgoing ECN field to whatever codepoint sits where the arriving inner header (row) and outer header (column) meet in Figure 4. (Also, the IPv4 header checksum changes any time the ECN field does 🧮.) There's no need for more than one decapsulation mode, because these rules cover every known requirement 💅.

            +---------+------------------------------------------------+
            |Arriving |            Arriving Outer Header               |
            |   Inner +---------+------------+------------+------------+
            |  Header | Not-ECT | ECT(0)     | ECT(1)     |     CE     |
            +---------+---------+------------+------------+------------+
            | Not-ECT | Not-ECT |Not-ECT(!!!)|Not-ECT(!!!)| <drop>(!!!)|
            |  ECT(0) |  ECT(0) | ECT(0)     | ECT(1)     |     CE     |
            |  ECT(1) |  ECT(1) | ECT(1) (!) | ECT(1)     |     CE     |
            |    CE   |      CE |     CE     |     CE(!!!)|     CE     |
            +---------+---------+------------+------------+------------+

The outgoing ECN field gets the codepoint at the crossing of the arriving inner header (row) and the arriving outer header (column), or the packet gets dropped where the table says so 🗑️. The combinations that aren't in use yet are marked (!!!) or (!).

Figure 4: New IP in IP Decapsulation Behaviour

This decapsulation table comes from the following logic 🧠:

  • If the inner ECN field is Not-ECT, the decapsulator MUST NOT propagate any other ECN codepoint onwards. The reason is that the inner Not-ECT marking comes from transports that use dropped packets as their congestion signal, so they wouldn't understand or respond to any other ECN codepoint [RFC4774]. Specifically:
    • If the inner ECN field is Not-ECT and the outer ECN field is CE, the decapsulator MUST drop the packet.
    • If the inner ECN field is Not-ECT and the outer ECN field is Not-ECT, ECT(0), or ECT(1), the decapsulator MUST forward the outgoing packet with the ECN field cleared to Not-ECT.
  • In all other cases where the inner supports ECN, the decapsulator MUST set the outgoing ECN field to the more severe marking of the outer and inner ECN fields. The ranking of severity from highest to lowest is CE, ECT(1), ECT(0), Not-ECT. This in no way precludes cases where ECT(1) and ECT(0) have the same severity.
  • Some combinations of inner and outer ECN fields can't result from any transition in any current or previous ECN tunnelling specification. These currently unused (CU) combinations are marked in Figure 4 with (!!!) or (!). The (!!!) ones are CU and always potentially dangerous, while the (!) ones are CU and possibly dangerous. In these cases, especially the more dangerous ones, the decapsulator SHOULD log the event, and it MAY also raise an alarm 🚨.
Just because the highlighted combinations are unused right now doesn't mean all the other combinations are always valid! Some are only valid if they came from a particular type of legacy ingress, and dangerous otherwise. So an implementation MAY let an operator configure logging and alarms for those additional header combinations that are known to be dangerous or CU for the particular tunnel endpoint setup deployed at run-time 👀.

Alarms SHOULD be rate-limited, so the weird combinations can't snowball into a flood of alarm messages 🌊. It MUST be possible to suppress alarms or logging. For example, if a combination that wasn't used before has started being used for legitimate purposes, like a new standards action, you'd want to be able to turn those off 🫶.

The logic above lets ECT(0) and ECT(1) both stand for the same level of congestion marking (for example, "not congestion marked"). It also leaves room for future schemes where ECT(1) is a more severe marking than ECT(0). That's especially handy for enabling the simplest possible encoding for PCN [PCN3in1] (see Section 5.3.2) 🦋. Whether ECT(1) is treated the same as ECT(0) or as a higher severity level is explained in the discussion of the ECN nonce [RFC3540] in Section 8, which in turn refers to Appendix D 📚.

4.3. Encapsulation Modes 🎀

Section 4.1 introduces two encapsulation modes, normal mode and compatibility mode, and defines how each one behaves for encapsulation (header copying or zeroing, respectively). Heads up that these are modes of the ingress tunnel endpoint only, not of the tunnel as a whole 🌸.

To comply with this specification, a tunnel ingress MUST at least implement normal mode. Unless it will never be used with legacy tunnel egress nodes (RFC 2003, RFC 2401, or RFC 2481 or the limited functionality mode of RFC 3168), an ingress MUST also implement compatibility mode, for backward compatibility with tunnel egresses that don't propagate explicit congestion notifications [RFC4774] 🕰️.

There are two ways an ingress tunnel endpoint can be paired with an egress: static or dynamically discovered 🔍.

Static: Tunnel endpoints are paired together by prior configuration.

Some encapsulator implementations are always deployed statically, and are constrained to never be paired with a legacy decapsulator (RFC 2003, RFC 2401, or RFC 2481 or the limited functionality mode of RFC 3168). In that case, only normal mode needs to be implemented 🧚.

Here's an example. IPsec tunnel endpoints compatible with RFC 4301 always use Internet Key Exchange Protocol version 2 (IKEv2) [RFC5996] for key exchange, and that protocol was introduced alongside RFC 4301. So both endpoints of an RFC 4301 tunnel can be sure the other end is compatible with RFC 4301, since the tunnel only forms after IKEv2 key management finishes, and at that point both ends are compliant with RFC 4301 by definition 💜. That means an IPsec tunnel ingress doesn't need compatibility mode, since it will never interact with legacy ECN tunnels. To comply with the present specification, it only needs to implement the required normal mode, which is identical to the pre-existing RFC 4301 behaviour ✨.

Dynamic Discovery: Tunnel endpoints are paired together by some form of tunnel endpoint discovery, typically finding an egress on the path taken by the first packet 🛰️.

This specification doesn't require or recommend dynamic discovery, and it doesn't define how dynamic negotiation might be done. It does recognise that proprietary tunnel endpoint discovery protocols exist, though, so it sets down some constraints on discovery protocols to keep interworking safe 🫶.

If dynamic tunnel endpoint discovery might pair an ingress with a legacy egress (RFC 2003, RFC 2401, or RFC 2481 or the limited functionality mode of RFC 3168), the ingress MUST implement both normal and compatibility mode. If the discovery process is arranged so it only ever finds a tunnel egress that propagates ECN (RFC 3168 full functionality mode, RFC 4301, or this present specification), then a tunnel ingress can be compliant with the present specification without implementing compatibility mode 🌈.

While a compliant tunnel ingress is discovering an egress, it MUST send packets in compatibility mode, in case the egress it finds turns out to be a legacy egress. If, through the discovery protocol, the egress indicates it's compliant with the present specification, with RFC 4301, or with RFC 3168 full functionality mode, the ingress can switch itself into normal mode. If the egress denies compliance with any of these, or returns an error implying it doesn't understand a request to work to any of these ECN specifications, the tunnel ingress MUST remain in compatibility mode 🙈.

If an ingress claims compliance with this specification, it MUST NOT permanently disable ECN processing across the tunnel (in other words, it can't only ever use compatibility mode). Sure, a tunnel ingress like that is at least safe with whatever ECN behaviour an egress has, but it misses the whole point of this specification, which is bringing ECN support to tunnels 😭.

So instead, if the ingress knows the egress does support propagation of ECN (full functionality mode of RFC 3168 or RFC 4301 or the present specification), it SHOULD use normal mode, so ECN gets supported wherever possible 💖. Quick note: this section started by saying an ingress "MUST implement" normal mode, and then it said an ingress "SHOULD use" normal mode. That difference is on purpose! It lets the mode be turned off in exceptional circumstances, but makes sure every implementation still makes normal mode available 🧐.

Implementation note: If a compliant node is the ingress for multiple tunnels, a mode setting will need to be stored for each tunnel ingress. However, if a node is the egress for multiple tunnels, none of those tunnels need to store a mode setting, because a compliant egress only needs one mode 🍓.

4.4. Single Mode of Decapsulation 🧚‍♀️

A compliant decapsulator only needs one mode of operation. But if a compliant egress is set up to be dynamically discoverable, it may need to respond to discovery requests from various types of legacy tunnel ingress. This specification doesn't define how dynamic negotiation might be done by (proprietary) discovery protocols, though it does set down some constraints to keep interworking safe 🔐.

Through the discovery protocol, a tunnel ingress compliant with the present specification might ask whether the egress is compliant with the present specification, with RFC 4301, or with RFC 3168 full functionality mode. Or an RFC 3168 tunnel ingress might try to negotiate for limited functionality or full functionality mode [RFC3168]. In all of these cases, a decapsulating tunnel egress compliant with this specification MUST agree to any of these requests, since it behaves identically in all of them 🤝.

If no ECN-related mode is requested, a compliant tunnel egress MUST continue without raising any error or warning, because its egress behaviour is compatible with all the legacy ingress behaviours that don't negotiate capabilities 🌷.

A compliant tunnel egress SHOULD raise a warning alarm about any requests to enter modes it doesn't recognise. But for 'forward compatibility' with standards actions that might be defined after it was implemented, it SHOULD continue operating 🌟.

5. Updates to Earlier RFCs 🎀

5.1. Changes to RFC 4301 ECN Processing ✨

Ingress: okay so an RFC 4301 IPsec encapsulator doesn't change at all with this spec 🫶. It just uses the normal mode, which encapsulates packets exactly the way RFC 4301 does 💅.

Egress: an RFC 4301 egress does need an update to the new decapsulation behaviour in Figure 4 to comply with this spec 🔥. The good news is it's backward compatible, so every inner and outer combo that any protocol in the RFC series has used so far is totally unaffected 🦋. Only combos that have never been used got changed. It's basically adding new behaviours to RFC 4301 decapsulation without messing with the existing ones, which is so respectful of the old code 🥹. The specific updates to Section 5.1.2 of RFC 4301 are:

  • The outer, not the inner, is propagated when the outer is ECT(1) and the inner is ECT(0) 💜;
  • A packet with Not-ECT in the inner and an outer of CE is dropped rather than forwarded as Not-ECT 🙈;
  • Certain combinations of inner and outer ECN field have been identified as currently unused. These can trigger logging and/or raise alarms 🚨.

Modes: RFC 4301 tunnel endpoints don't need modes and aren't updated by the modes in this spec 🌸. Basically, an RFC 4301 IPsec ingress only ever uses the REQUIRED normal mode of encapsulation, which is unchanged from RFC 4301 encapsulation. It will never need the OPTIONAL compatibility mode explained in Section 4.3 🧚.

5.2. Changes to RFC 3168 ECN Processing 🎀

Ingress: on encapsulation, the new rule in Figure 3, that a normal mode tunnel ingress copies any ECN field into the outer header, updates the full functionality behaviour of an RFC 3168 ingress (Section 9.1.1 of [RFC3168]) ✨. Still, the new compatibility mode encapsulates packets identically to the limited functionality mode of an RFC 3168 ingress, so nothing weird happens there 🍓.

Egress: an RFC 3168 egress will need to be updated to the new decapsulation behaviour in Figure 4 to comply with this spec 💖. Again, it's backward compatible, so combos from any protocol in the RFC series so far are unaffected 🌈. Only never-used combos got changed, which adds new behaviours to RFC 3168 decapsulation without changing the existing ones 👑. The specific updates to Section 9.1.1 of RFC 3168 are:

  • The outer, not the inner, is propagated when the outer is ECT(1) and the inner is ECT(0) 💜;
  • Certain combinations of inner and outer ECN field have been identified as currently unused. These can trigger logging and/or raise alarms 🚨.

Modes: an RFC 3168 ingress will need to be updated if it's going to comply with this spec, whether or not it implemented the optional full functionality mode of Section 9.1.1 of RFC 3168 🫠.

Section 9.1 of RFC 3168 defined a (required) limited functionality mode and an (optional) full functionality mode for a tunnel 🧃. In RFC 3168, modes applied to both ends of the tunnel, but in this spec, modes are only used at the ingress -- a single egress behaviour covers all cases 🦋.

The normal mode of encapsulation is an update to the encapsulation behaviour of the full functionality mode of an RFC 3168 ingress ✨. The compatibility mode of encapsulation is identical to the encapsulation behaviour of the limited functionality mode of an RFC 3168 ingress, except it is not always obligatory 🌸.

The constraints on how tunnel discovery protocols set modes in Sections 4.3 and 4.4 are an update to RFC 3168 💅, but they're unlikely to need code changes since they document existing safe practice 🫶.

5.3. Motivation for Changes 💭

An overriding goal here is to make sure the same ECN signals mean the same thing no matter which tunnels happen to encapsulate an IP packet flow 🌈. This removes gratuitous inconsistency, which otherwise constrains the available design space and makes it harder to design networks and new protocols that work predictably 😭.

5.3.1. Motivation for Changing Encapsulation 🧚

The normal mode in Section 4 updates RFC 3168 so that all IP-in-IP encapsulation of the ECN field is consistent 💜. It's consistent with the way both RFC 4301 IPsec [RFC4301] and IP-in-MPLS or MPLS-in-MPLS encapsulation [RFC5129] construct the ECN field 🔥.

Compatibility mode was also defined so that an ingress compliant with a version of IPsec prior to RFC 4301 can still switch to using drop across a tunnel 🫶. That keeps it backward compatible with legacy decapsulators that don't propagate ECN 🍓.

The trigger for this update to RFC 3168 encapsulation was a Standards-Track proposal for pre-congestion notification (PCN [RFC5670]) 🦋. PCN excess-traffic-marking only works correctly if the ECN field is copied on encapsulation (as in RFC 4301 and RFC 5129). It does not work if ECN is reset (as in RFC 3168) 🙈. That's because PCN excess-traffic-marking depends on the outer header revealing any congestion experienced so far on the whole path, not just since the last tunnel ingress 💀.

PCN lets a network operator add flow admission and termination for inelastic traffic at the edges of a Diffserv domain 🌸. It does this without any per-flow mechanisms in the interior and without the generous provisioning typical of Diffserv, aiming to significantly reduce costs 💰. The PCN architecture [RFC5559] states that RFC 3168 IP-in-IP tunnelling of the ECN field cannot be used for any tunnel ingress in a PCN domain 😬. Before this spec, that left a stark choice: either you couldn't use PCN for inelastic traffic control, or you couldn't use the many tunnels already deployed for Mobile IP, VPNs, and so forth 😭.

This spec provides a clean solution to that problem 👑. Network operators who want to use both PCN and tunnels can specify that every tunnel ingress in a PCN region must comply with this latest specification ✨.

Rather than let tunnel specifications fragment further into one for PCN, one for IPsec, and one for other tunnels, the opportunity was taken to consolidate the diverging specs back into a single tunnelling behaviour 🎀. Resetting ECN was originally motivated by a covert channel concern that has been deliberately set aside in RFC 4301 IPsec 🧠. So the reset behaviour of RFC 3168 is an anomaly we don't need to keep 🫠. Copying ECN on encapsulation is simpler than resetting 💅. As more tunnel endpoints comply with this single consistent specification, encapsulation will be simpler as well as more predictable 🌈.

Appendix B assesses whether copying rather than resetting CE on ingress will cause any unintended side effects, from three perspectives: security, control, and management 🔍. In summary, that analysis finds that:

  • From the control perspective, either copying or resetting works for existing arrangements. But copying has more potential for simplifying control, and resetting breaks at least one proposal that's already on the Standards Track 🙈;
  • From the management and monitoring perspective, copying is preferable 💜;
  • From the traffic security perspective (enforcing congestion control, mitigating denial of service, etc.), copying is preferable 🛡️;
  • From the information security perspective, resetting is preferable. However, the IETF Security Area now considers copying acceptable, given the bandwidth of a two-bit covert channel can be managed 🔐.

So, there are two points against resetting CE on ingress, while copying CE causes no significant harm 🦋✨.

5.3.2. Motivation for Changing Decapsulation 💅

okay bestie, so the decapsulation rules in Section 4 fix three problems with what RFC 3168 and RFC 4301 both did before 🫶

  1. okay so the old rules blocked alternate ECN meanings that could signal more than one severity level of congestion [RFC4774], [RFC5559]. the two-bit ECN field has four states, which is enough room for two severity levels on top of not-congested and not-ECN-capable 🌈 but the old rules assumed ECT(0) and ECT(1) are always equivalent. that wastes one of the four codepoints (half a bit) in the IP (v4 and v6) header 😭 the new rules work either way, whether ECT(1) is more severe than or equivalent to ECT(0) ✨

as explained in Appendix B.1, the original reason for not forwarding the outer ECT codepoints was to keep the covert channel across a decapsulator to 1 bit per packet. but now the IETF Security Area has decided that a two-bit covert channel through an encapsulator is a manageable risk, so the same logic should hold for a decapsulator too 🦋

this is also good for general future-proofing, but it's immediately urgent for pre-congestion notification (PCN), which uses two severity levels of congestion 🔥 if a congested queue put ECT(1) in the outer header to signal more severe congestion than ECT(0), the old decapsulation rules would have thrown that signal away 💀 and tunnelled traffic would never know it should reduce its load. the end 🙅‍♀️

before this spec was written, the PCN working group had to juggle a bunch of wasteful or convoluted work-rounds for this. no shade to the ingenuity, but none were picked for the Standards Track because they were somewhat wasteful, imprecise, or complicated 🍓 instead, a baseline PCN encoding was specified [RFC5696] that only supported one severity level of congestion, but left room for those work-rounds as experimental extensions.

the simplest approach, and honestly the iconic one, is the one this spec takes: just remove the covert channel blockages from tunnelling behaviour, since they're now deemed unnecessary anyway 👑 then network operators who want two congestion severity levels for PCN can specify that every tunnel egress in a PCN region must comply with this latest spec. once that's done, the simplest possible encoding for PCN with two severity levels of congestion [PCN3in1] can be used 🧚

and it's not just PCN that wins here. two congestion severity levels become available for other potential uses of the extra ECN codepoint too (e.g., [VCP]) 🌸

  1. ngl this one is very real: there are documented cases where a middlebox (e.g., a firewall) drops packets with header values that were currently unused (CU) when the box was deployed, usually because anything unexpected might be an attack 🙈 that tends to bar future use of CU values. the new decapsulation rules specify optional logging and/or alarms for specific combinations of inner and outer headers that are currently unused. the aim is to give implementers a recourse other than drop if they're worried about the security of CU values 💖 it respects legitimate security concerns about CU values, but still makes their future use easier.

if the alarms get interpreted as an attack (e.g., by a management system), the offending packets can be dropped. however, alarms can be turned off once these combinations come into regular use (e.g., through a future standards action) ✨

  1. while reviewing the currently unused combinations of inner and outer headers, we also took the chance to define one consistent behaviour for the three cases with a Not-ECT inner header but a different outer one. RFC 3168 and RFC 4301 had drifted apart on this, and even their shared behaviours had never been justified 😭

none of these combinations should come out of Internet protocols in the RFC series, but future standards actions might find good uses for any or all of them 🌈 so the decision was that a decapsulator must forward a Not-ECT inner header unchanged when the arriving outer header is ECT(0) or ECT(1). for safety, it must drop a combination of Not-ECT inner and CE outer headers 🛡️ then, if some unfortunate misconfiguration had a congested router marking CE on a packet that started out Not-ECT, drop would be the only appropriate signal for the egress to pass along. it's the only signal a non-ECN-capable transport (Not-ECT) would understand 🫠

it might seem contradictory that the same reasoning wasn't applied to the ECT(1) codepoint, given it's being proposed as an intermediate level of congestion in a scheme moving through the IETF [PCN3in1]. but a decapsulator must forward a Not-ECT inner unchanged when its outer is ECT(1) 🦋 the reason for not dropping this CU combination is to make sure it stays usable if it's needed in the future. if any misconfiguration led to ECT(1) congestion signals with a Not-ECT inner, it wouldn't be disastrous for the tunnel egress to suppress them, because the congestion should then escalate to CE marking, which the egress would drop, which at least prevents congestion collapse 💜

Problems 2 and 3 alone wouldn't warrant a change to decapsulation, but it was decided they're worth fixing and making consistent at the same time as the decapsulation code is changed to fix problem 1 (two congestion severity levels) 🎀

6. Backward Compatibility ✨

okay, the big question: does this spec play nicely with older tunnelling RFCs? yes, bestie 🥹 a tunnel endpoint compliant with the present specification is backward compatible when paired with any tunnel endpoint compliant with any previous tunnelling RFC, whether RFC 4301, RFC 3168 (see Section 3), or the earlier RFCs summarised in Appendix A (RFC 2481, RFC 2401, and RFC 2003). each case is enumerated below 💅

6.1. Non-Issues Updating Decapsulation 🌸

at the egress, this spec only augments the per-packet calculation of the ECN field (RFC 3168 and RFC 4301) for combinations of inner and outer headers that haven't been used in any IETF protocols so far. so basically nothing that's already happening gets disturbed 🫶

therefore, all other things being equal, if an RFC 4301 IPsec egress is updated to comply with the new rules, it will still interwork with any ingress compliant with RFC 4301, and the packet outputs will be identical to what it would have output before (fully backward compatible) 💖

and, all other things being equal, if an RFC 3168 egress is updated to comply with the same new rules, it will still interwork with any ingress complying with any previous specification (both modes of RFC 3168, both modes of RFC 2481, RFC 2401, and RFC 2003), and the packet outputs will be identical to those it would have output before (fully backward compatible) 🧚

a compliant tunnel egress just needs to implement the one behaviour in Section 4, with no additional mode or option configuration at the ingress or egress, and no additional negotiation with the ingress 🙌 the new decapsulation rules are defined so that congestion control still works safely if any of the earlier versions of ECN processing get used unilaterally at the encapsulating ingress of the tunnel (any of RFC 2003, RFC 2401, either mode of RFC 2481, either mode of RFC 3168, RFC 4301, and this present specification) 🔥

6.2. Non-Update of RFC 4301 IPsec Encapsulation 🎀

an RFC 4301 IPsec ingress can comply with this new specification without any update, and it doesn't need any new modes, options, or configuration. so, all other things being equal, it will keep interworking identically with any egress it worked with before (fully backward compatible) 🦋

6.3. Update to RFC 3168 Encapsulation 🍓

the encapsulation behaviour of the new normal mode copies the ECN field, whereas an RFC 3168 ingress in full functionality mode reset it. however, all other things being equal, if an RFC 3168 ingress is updated to the present specification, the outgoing packets from any tunnel egress will still be unchanged 🥰 that's because every variant of tunnelling at either end (RFC 4301, both modes of RFC 3168, both modes of RFC 2481, RFC 2401, RFC 2003, and the present specification) has always carried an incoming CE marking through the inner header and onward into the outgoing header, whether the outer header is reset or copied 💜

so if the tunnel is treated as a black box, the packets output from any egress will be identical with or without an update to the ingress. but if you peek inside the black box (between the tunnel endpoints), CE markings copied by the updated ingress will be visible there, which they wouldn't have been before 👀 so the update to encapsulation can be termed 'black-box backward compatible' (i.e., identical unless you look inside the tunnel) ✨

this spec doesn't add any new backward compatibility issues when a compliant ingress talks with a legacy egress, but it still has to provide similar safeguards to the ones RFC 3168 already defined. RFC 3168 laid down rules so that an RFC 3168 ingress turns off ECN (limited functionality mode) when it's paired with a legacy egress (RFC 2481, RFC 2401, or RFC 2003), since those wouldn't propagate ECN correctly 🙈 the present specification carries those rules forward (Section 4.3). it uses compatibility mode whenever RFC 3168 would have used limited functionality mode, and their per-packet behaviours are identical 🫶 so, all other things being equal, an ingress using the new rules will interwork with any legacy tunnel egress in exactly the same way as an RFC 3168 ingress (still black-box backward compatible) 💅🌈

7. Design Principles for Alternate ECN Tunnelling Semantics ✨

This section is informative, not normative. 🫶 Translation: it's advice, not a rulebook, bestie.

Okay so, Section 5 of RFC 3168 lets the Diffserv codepoint (DSCP)[RFC2474] "switch in" alternative behaviours for marking the ECN field, just like it switches in different per-hop behaviours (PHBs) for scheduling 🧚. [RFC4774] gives best current practice for designing these alternative ECN semantics, and it only briefly mentions in Section 5.4 that tunnelling needs to be considered 💀. So the guidance below is the extra help: it complements and extends RFC 4774 for anyone designing alternate ECN semantics that also need alternate tunnelling semantics 🎀.

The overriding guidance is: "Avoid designing alternate ECN tunnelling semantics, if at all possible" 🚫✨. Here's why it's giving chaos. If a scheme needs tunnels to do special processing of the ECN field for certain DSCPs, it gets really hard to guarantee that every implementer of every tunnel added the required exception, or that operators have ubiquitously deployed the required updates 😭. A single authority is unlikely to even know about all the tunnels in a network, which can include tunnels that applications set up between endpoints, or tunnels created dynamically in the network 🙈. So it's highly likely that some tunnels within a network, or on hosts connected to it, won't implement the required special case. Not ideal, periodt.

That said, if a non-default scheme for tunnelling the ECN field is really required, the following guidelines might prove useful in its design 💅.

On encapsulation in any alternate scheme:

  1. The ECN field of the outer header ought to be cleared to Not-ECT ("00") unless it is guaranteed that the corresponding tunnel egress will correctly propagate congestion markings introduced across the tunnel in the outer header 🌸.
  2. If it has established that ECN will be correctly propagated, an encapsulator also ought to copy incoming congestion notification into the outer header. The general principle here is that the outer header should reflect congestion accumulated along the whole upstream path, not just since the tunnel ingress (Appendix B.3 on management and monitoring explains) 🦋.

In some circumstances (e.g., PCN [RFC5559] and perhaps some pseudowires [RFC5659]), the whole path is divided into segments, each with its own congestion notification and feedback loop 🌈. In these cases, the function that regulates load at the start of each segment will need to reset congestion notification for its segment. Often, the point where congestion notification is reset will also be located at the start of a tunnel. However, the resetting function can be thought of as being applied to packets after the encapsulation function, so these are two logically separate functions even though they might run on the same physical box 🧠. Then, the code module doing encapsulation can keep to the copying rule and the load regulator module can reset congestion, without any code in either module being conditional on whether the other is there. Main character energy for modular code 👑.

On decapsulation in any alternate scheme:

  1. If the arriving inner header is Not-ECT, the transport will not understand other ECN codepoints. If the outer header carries an explicit congestion marking, the alternate scheme would be expected to drop the packet, since that's the only indication of congestion the transport will understand 😤. If the alternate scheme recommends forwarding rather than dropping such a packet, it will need to clearly justify this decision. If the inner is Not-ECT and the outer carries any other ECN codepoint that does not indicate congestion, the alternate scheme can forward the packet, but probably only as Not-ECT 🍓.
  2. If the arriving inner header is one other than Not-ECT, the ECN field that the alternate decapsulation scheme forwards ought to reflect the more severe congestion marking of the arriving inner and outer headers ⭐.
  3. Any alternate scheme will need to define a behaviour for all combinations of inner and outer headers. That includes combinations that wouldn't be expected from the standards known at the time, and even ones that wouldn't be expected from the tunnel ingress paired with the egress at run-time 🧚‍♀️. Consideration should be given to logging such unexpected combinations and raising an alarm, particularly if there's a danger that the invalid combination implies congestion signals aren't being propagated correctly 🚨. The presence of currently unused combinations may represent an attack, but the new scheme should try to define a way to forward such packets, at least if a safe outgoing codepoint can be defined 💜.

Raising an alarm allows a management system to decide whether the anomaly is indeed an attack, in which case it can decide to drop such packets 🫠. This is a preferable approach to hard-coded discard of packets that seem anomalous today but may be needed tomorrow in future standards actions. Honestly, that's the most mature take in the whole doc 💅✨.

8. Security Considerations 🔐

Appendix B.1 discusses the security constraints imposed on ECN tunnel processing 🕵️‍♀️. The new rules for ECN tunnel processing (Section 4) trade off between information security (covert channels) and traffic security (congestion monitoring and control). Ensuring congestion markings are not lost is itself an aspect of security, because if we allowed congestion notification to be lost, any attempt to enforce a response to congestion would be much harder 😩.

Security issues in unlikely, but possible, scenarios:

  • Tunnels intersecting Diffserv regions with alternate ECN semantics: If alternate congestion notification semantics are defined for a certain Diffserv PHB, the scope of the alternate semantics might typically be bounded by the limits of a Diffserv region or regions, as envisaged in [RFC4774] (e.g., the pre-congestion notification architecture [RFC5559]). The inner headers in tunnels crossing the boundary of such a Diffserv region but ending within the region can potentially leak the external congestion notification semantics into the region, or leak the internal semantics out of the region. [RFC2983] discusses the need for Diffserv traffic conditioning to be applied at these tunnel endpoints as if they are at the edge of the Diffserv region. Similar concerns apply to any processing or propagation of the ECN field at the endpoints of tunnels with one end inside and the other outside the domain. [RFC5559] gives specific advice on this for the PCN case, but other definitions of alternate semantics will need to discuss the specific security implications in each case 🚧.
  • ECN nonce tunnel coverage: The new decapsulation rules improve the coverage of the ECN nonce [RFC3540] relative to the previous rules in RFC 3168 and RFC 4301. However, nonce coverage is still not perfect, as this would have led to a safety problem in another case. Both are corner-cases, so discussion of the compromise between them is deferred to Appendix D 🥲.
  • Covert channel not turned off: A legacy (RFC 3168) tunnel ingress could ask an RFC 3168 egress to turn off ECN processing as well as itself turning off ECN. An egress compliant with the present specification will agree to such a request from a legacy ingress, but it relies on the ingress always sending Not-ECT in the outer header. If the egress receives other ECN codepoints in the outer, it will process them as normal, so it will actually still copy congestion markings from the outer to the outgoing header. Referring, for example, to Figure 5 (Appendix B.1), although the tunnel ingress 'I' will set all ECN fields in outer headers to Not-ECT, 'M' could still toggle CE or ECT(1) on and off to communicate covertly with 'B', because we have specified that 'E' only has one mode regardless of what mode it says it has negotiated 👀. We could have specified that 'E' should have a limited functionality mode and check for such behaviour. However, we decided not to add the extra complexity of two modes on a compliant tunnel egress merely to cater for an historic security concern that is now considered manageable 🫶.

9. Conclusions 🎀

This document allows tunnels to propagate an extra level of congestion severity 📈. It uses previously unused combinations of inner and outer headers to augment the rules for calculating the ECN field when decapsulating IP packets at the egress of IPsec (RFC 4301) and non-IPsec (RFC 3168) tunnels 🔥.

This document also updates the ingress tunnelling encapsulation of RFC 3168 ECN to bring all IP-in-IP tunnels into line with the new behaviour in the IPsec architecture of RFC 4301, which copies rather than resets the ECN field when creating outer headers 🧃.

The need for both these updated behaviours was triggered by the introduction of pre-congestion notification (PCN) onto the IETF Standards Track 🌟. Operators wanting to support PCN or other alternate ECN schemes that use an extra severity level can require that their tunnels comply with the present specification. This is not a fork in the RFC series, it is an update that can be deployed first by those that need it, and subsequently by all tunnel endpoint implementations during general code maintenance 🦋. It is backward compatible with all previous tunnelling behaviours, so existing single severity level schemes will continue to work as before, but support for two severity levels will gradually be added to the Internet 🌈.

The new rules propagate changes to the ECN field across tunnel endpoints that previously blocked them to restrict the bandwidth of a potential covert channel. Limiting the channel's bandwidth to two bits per packet is now considered sufficient ✨.

At the same time as removing these legacy constraints, the opportunity has been taken to draw together diverging tunnel specifications into a single consistent behaviour 💖. Then, any tunnel can be deployed unilaterally, and it will support the full range of congestion control and management schemes without any modes or configuration. Further, any host or router can expect the ECN field to behave in the same way, whatever type of tunnel might intervene in the path. This new certainty could enable new uses of the ECN field that would otherwise be confounded by ambiguity 🔮.

10. Acknowledgements 🫶

Thanks to David Black for his insightful reviews and patient explanations of better ways to think about function placement and alarms 🙌. Thanks to David and to Anil Agarwal for pointing out cases where it is safe to forward CU combinations of headers 💜. Also, thanks to Arnaud Jacquet for the idea for Appendix C 🍓. Thanks to Gorry Fairhurst, Teco Boot, Michael Menth, Bruce Davie, Toby Moncaster, Sally Floyd, Alfred Hoenes, Gabriele Corliano, Ingemar Johansson, Philip Eardley, and David Harrington for their thoughts and careful review comments, and to Stephen Hanna, Ben Campbell, and members of the IESG for respectively conducting the Security Directorate, General Area, and IESG reviews 🎉.

Bob Briscoe is partly funded by Trilogy, a research project (ICT-216372) supported by the European Community under its Seventh Framework Programme 🇪🇺✨.

11. References 📚

11.1. Normative References

okay bestie, these are the must-reads 📚🎀

  • [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, October 1996.
  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, September 2001.
  • [RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005.

11.2. Informative References

these ones are the extra reading for when you're obsessed and want more 🧚💜

  • [PCN3in1] Briscoe, B., Moncaster, T., and M. Menth, "Encoding 3 PCN-States in the IP header using a single DSCP", Work in Progress, July 2010.
  • [RFC2401] Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998.
  • [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998.
  • [RFC2481] Ramakrishnan, K. and S. Floyd, "A Proposal to add Explicit Congestion Notification (ECN) to IP", RFC 2481, January 1999.
  • [RFC2983] Black, D., "Differentiated Services and Tunnels", RFC 2983, October 2000.
  • [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit Congestion Notification (ECN) Signaling with Nonces", RFC 3540, June 2003.
  • [RFC4774] Floyd, S., "Specifying Alternate Semantics for the Explicit Congestion Notification (ECN) Field", BCP 124, RFC 4774, November 2006.
  • [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion Marking in MPLS", RFC 5129, January 2008.
  • [RFC5559] Eardley, P., "Pre-Congestion Notification (PCN) Architecture", RFC 5559, June 2009.
  • [RFC5659] Bocci, M. and S. Bryant, "An Architecture for Multi-Segment Pseudowire Emulation Edge-to-Edge", RFC 5659, October 2009.
  • [RFC5670] Eardley, P., "Metering and Marking Behaviour of PCN-Nodes", RFC 5670, November 2009.
  • [RFC5696] Moncaster, T., Briscoe, B., and M. Menth, "Baseline Encoding and Transport of Pre-Congestion Information", RFC 5696, November 2009.
  • [RFC5996] Kaufman, C., Hoffman, P., Nir, Y., and P. Eronen, "Internet Key Exchange Protocol Version 2 (IKEv2)", RFC 5996, September 2010.
  • [VCP] Xia, Y., Subramanian, L., Stoica, I., and S. Kalyanaraman, "One more bit is enough", Proc. SIGCOMM'05, ACM CCR 35(4)37--48, 2005, <http://doi.acm.org/10.1145/1080091.1080098>.

Appendix A. Early ECN Tunnelling RFCs ✨

IP-in-IP tunnelling started out in [RFC2003]. When encapsulating, the inner header got copied to the outer one, and when decapsulating, the outer one just got thrown away. IPsec tunnelling [RFC2401] initially did the exact same thing, so it's giving déjà vu 💅

Then ECN showed up experimentally in [RFC2481], and here's the tea: legacy tunnels (RFC 2003 or RFC 2401) would have tossed any congestion markings added to the outer header 😭. So RFC 2481 brought in rules for working out the outgoing header from a mix of the inner and outer headers on decapsulation. It also added a second mode for IPsec tunnels. In that mode, ECN processing got switched off (Not-ECT) in the outer header on encapsulation, because an RFC 2401 decapsulator would discard the outer header anyway. For RFC 2401 IPsec, this also had the side effect of completely blocking the covert channel 🙈

In RFC 2481, the ECN field was two separate bits. But when ECN moved from Experimental to Standards Track [RFC3168], the field got redefined as four codepoints. That meant decapsulation needed a different calculation from the one RFC 2481 used, which is a whole glow-up 🦋 RFC 3168 also had two modes. One was a 'full functionality mode' that restricted the covert channel as much as possible but still let ECN work with IPsec. The other completely turned off ECN processing across the tunnel. This 'limited functionality mode' gave operators a way to fully block the covert channel, and it let an RFC 3168 ingress interwork with a legacy tunnel egress (RFC 2481, RFC 2401, or RFC 2003). Iconic, honestly 👑

The present specification includes a similar compatibility mode so it can interwork safely with tunnels that follow any of these three earlier RFCs. The difference is that, unlike RFC 3168, this is only a mode of the ingress, because decapsulation behaves the same either way ✨

Appendix B. Design Constraints 🌈

Tunnel processing of a congestion notification field has to meet congestion control and management needs without creating new information security vulnerabilities (if information security is required). This appendix documents the analysis of the trade-offs between these factors that led to the new encapsulation rules in Section 4.1. It's giving careful engineering, bestie 🫶

B.1. Security Constraints

Information security can be assured by using various end-to-end security solutions (including IPsec in transport mode [RFC4301]), but a commonly used scenario involves communicating between two physically protected domains across the public Internet. In this case, there are certain management advantages to using IPsec in tunnel mode only across the publicly accessible part of the path. The path then crosses security 'domains': the ones protected by physical or other means before and after the tunnel, and the one protected by an IPsec tunnel across the otherwise unprotected domain. The scenario in Figure 5 is used, where endpoints 'A' and 'B' communicate through a tunnel. The tunnel ingress 'I' and egress 'E' sit within physically protected edge domains, while the tunnel spans an unprotected internetwork where there may be 'men in the middle', M.

            physically       unprotected     physically
        <-protected domain-><--domain--><-protected domain->
        +------------------+            +------------------+
        |                  |      M     |                  |
        |    A-------->I=========>==========>E-------->B   |
        |                  |            |                  |
        +------------------+            +------------------+
                       <----IPsec secured---->
                               tunnel

                  Figure 5: IPsec Tunnel Scenario

IPsec encryption is typically used to stop 'M' from seeing messages from 'A' to 'B'. IPsec authentication stops 'M' from masquerading as the sender of messages from 'A' to 'B' or altering their contents. 'I' can use IPsec tunnel mode to let 'A' communicate with 'B', but impose encryption so 'A' doesn't leak information to 'M'. Or 'E' can insist that 'I' uses tunnel mode authentication so 'M' can't communicate information to 'B' 🕵️‍♀️

Mutable IP header fields like the ECN field (plus the Time to Live (TTL) / Hop Limit and DS fields) can't be included in the cryptographic calculations of IPsec. So if 'I' copies these mutable fields into the outer header that's exposed across the tunnel, it will have allowed a covert channel from 'A' to 'M' that bypasses its encryption of the inner header. And if 'E' copies these fields from the outer header to the outgoing one, even if it validates authentication from 'I', it will have allowed a covert channel from 'M' to 'B' 😭

ECN at the IP layer is designed to carry information about congestion from a congested resource towards downstream nodes. Typically, a downstream transport might feed the information back somehow to the point upstream of the congestion that can regulate the load on the congested resource, but other actions are possible [RFC3168], Section 6. In terms of the unicast scenario above, ECN effectively intends to create an information channel (for congestion signalling) from 'M' to 'B' (for 'B' to feed back to 'A'). Therefore, the goals of IPsec and ECN are mutually incompatible, which means some compromise is needed 🥺

With respect to using the DS or ECN fields as covert channels, Section 5.1.2 of RFC 4301 says, "controls are provided to manage the bandwidth of this channel". Using the ECN processing rules of RFC 4301, the channel bandwidth is two bits per datagram from 'A' to 'M' and one bit per datagram from 'M' to 'B' (because 'E' limits the combinations of the 2-bit ECN field that it will copy). In both cases, the covert channel bandwidth is further reduced by noise from any real congestion marking. RFC 4301 implies that these covert channels are limited enough to be considered a manageable threat. However, for the larger (six-bit) DS field, the same section of RFC 4301 says not copying is the default, but a configuration option can allow copying "to allow a local administrator to decide whether the covert channel provided by copying these bits outweighs the benefits of copying". Of course, an administrator who plans to copy the DS field has to take into account that it could be concatenated with the ECN field, creating a covert channel with eight bits per datagram 💀

For tunnelling the six-bit Diffserv field, two conceptual models had to be defined so administrators can trade off security against the needs of traffic conditioning [RFC2983]:

  • The uniform model: where the Diffserv field is preserved end-to-end by copying it into the outer header on encapsulation and copying it from the outer header on decapsulation.
  • The pipe model: where the outer header is independent of the one in the inner header, so it hides the Diffserv field of the inner header from any interaction with nodes along the tunnel.

However, for ECN, the new IPsec security architecture in RFC 4301 only standardised one tunnelling model, which is equivalent to the uniform model. It decided that simplicity mattered more than giving administrators the option of a tiny increment in security, especially since not copying congestion indications could seriously harm everyone's network service 🍓

B.2. Control Constraints 🎀

Okay so congestion control has one big requirement: any congestion notification that a resource marks into packets has to be able to travel back through a feedback loop to something that can actually control the load on that resource. Just to be precise, we're calling that thing the 'Load Regulator' instead of "the data source," because sometimes load isn't regulated by the data source. Usually the two mean the same thing though 💅. Also, "a function _capable of_ controlling the load" deliberately covers a source app that doesn't actually control the load but honestly should, like an app with no congestion control that runs over UDP 🙈.

                 A--->R--->I=========>M=========>E-------->B

                     Figure 6: Simple Tunnel Scenario

Now we're looking at a tunnel setup like the IPsec one we just covered, minus the different security domains, because this time the focus is on whether the control loop and management monitoring actually work 🦋. If resources in the tunnel are going to explicitly notify congestion, and the feedback path runs from 'B' to 'A', then 'E' has to copy any CE marking from the outer header to the outgoing header so it can get to 'B'. Otherwise congestion notification from resources like 'M' can't get fed back to the Load Regulator ('A') 😭. But it doesn't seem necessary for 'I' to copy CE markings from the incoming header to the outer header. For example, if resource 'R' is congested, it can send that info to 'B' using the congestion field in the inner header, with no need for 'I' to copy it into the outer header and for 'E' to copy it back out. And 'E' can still write any extra congestion marking picked up across the tunnel into the congestion field of the outgoing header ✨.

So basically, 'E' can keep the control loop working whether 'I' copies congestion notification into the outer header or resets it. Iconic 👑.

That's how existing control setups work. But copying reveals more information, so it could open the door to better control system designs 🧚. For instance, resetting CE marking on encapsulation breaks the Standards-Track PCN congestion marking scheme [RFC5670]. It ends up removing way more traffic than it needs to (Section 5.3.1) 💀. Copying CE markings at ingress, on the other hand, gives the correct control behaviour 🫶.

B.3. Management Constraints 🍓

Control isn't the only thing we have to care about. There are management constraints too 🌸. A management system might monitor congestion markings on packets as they pass by, maybe at the border between networks as part of a service level agreement. For example, monitors at the borders of autonomous systems may need to measure how much congestion has built up so far along the path, maybe to work out how much of it each domain contributed 🔥.

In this document, the baseline of congestion marking (the Congestion Baseline) is defined as the source of the layer that created, or most recently reset, the congestion notification field 💜. When you're monitoring congestion, it'd be nice if the Congestion Baseline didn't change depending on whether packets were tunnelled. And since some tunnels cross domain borders (say 'M' in Figure 6 is monitoring a border), it would be desirable for 'I' to copy the congestion accumulated so far into the outer headers, so it gets exposed across the tunnel 🌈.

For management purposes, it might also be useful for the tunnel egress to tell whether congestion happened across a tunnel or upstream of it 🎀. At first glance, copying congestion markings at the ingress seems like it would make that harder, while it was easy when an RFC 3168 ingress reset them. But Appendix C gives a simple and precise method for a tunnel egress to infer the congestion level introduced across a tunnel. It works whether the ingress copies or resets congestion markings, so no stress 🦋.

Appendix C. Contribution to Congestion across a Tunnel 🔍

This specification says that a tunnel ingress determines the ECN field of each new outer tunnel header by copying the arriving header. People have worried this will make it hard for the tunnel egress to monitor congestion introduced only along a tunnel. That's easy if the outer ECN field gets reset at the tunnel ingress (RFC 3168 full functionality mode). But copying CE marks at ingress still makes it easy for the egress to measure congestion introduced across a tunnel, and here's how 💖.

Say the egress looks at 100 packets. It measures that 30 are CE marked in both the inner and outer headers, and 12 have extra CE marks in the outer header but not the inner one. That means packets arriving at the ingress had already seen 30% congestion. But it does NOT mean there was 12% congestion across the tunnel 🙈. The right calculation is p_t = 12/(100-30) = 12/70 = 17%. And that's easy for the egress to measure: it's just the proportion of packets that aren't marked in the inner header (70) that do have a CE marking in the outer header (12) 🫶. This technique works whether the ingress copies or resets CE markings, so an egress that isn't sure which RFC the ingress complies with can still use it ✨.

Figure 7 shows this as a combinatorial probability diagram 🧠. The square is 100 packets. The 30% divide along the bottom is marking before the ingress, and the p_t divide up the side is marking introduced across the tunnel 🌸.

     ^ outer header marking
     |
100% +-----+---------+       The large square
     |     |         |       represents 100 packets
     | 30  |         |
     |     |         |   p_t = 12/(100-30)
 p_t +     +---------+       = 12/70
     |     |   12    |       = 17%
   0 +-----+---------+--->
     0    30%       100%  inner header marking

    Figure 7: Tunnel Marking of Packets Already Marked at Ingress

Appendix D. Compromise on Decap with ECT(1) Inner and ECT(0) Outer 🤔

A packet with an ECT(1) inner and an ECT(0) outer should never come from any known IETF protocol 🙅‍♀️. RFC 3168 and RFC 4301 both say to ignore the outer when decapsulating a packet like that, but they don't give a reason. This appendix explains why we decided not to change that advice 💅.

The short version: ECT(0) always means 'not congested', and ECT(1) may mean the same thing [RFC3168], or it may mean a higher severity congestion signal [RFC4774], [PCN3in1], depending on which transport is in use. Whether or not they mean the same thing, at the ingress the outer should have started out the same as the inner. Only a broken or compromised router could have changed the outer to ECT(0) 😭.

The decapsulator can detect this anomaly. The real question is whether it should fix the anomaly by ignoring the outer, or reveal the anomaly to the end-to-end transport by forwarding the outer 🤷‍♀️.

After weighing it up, we decided the decapsulator should fix the anomaly but log the event and optionally raise an alarm ✨. That's the safe move if ECT(1) is being used as a more severe marking than ECT(0), because it passes the more severe signal up to the transport. But hiding anomalies isn't a good look, which is why an optional alarm is suggested 🙈. It's also worth noting this anomaly could come from two changes to the outer header: a broken or compromised router inside the tunnel might be erasing congestion markings that a congested router introduced earlier in the same tunnel 💀. In that case the anomaly would be losing congestion signals, which needs immediate attention 🚨.

The original reason for defining ECT(0) and ECT(1) as equivalent was so the data source could use the ECN nonce [RFC3540] to detect whether congestion signals were being erased. But here the decapsulator doesn't need a nonce to spot anomalies introduced inside the tunnel, because it has the inner header as a record of what the header looked like at the ingress 🧚. So we decided the best compromise was to put solving the safety issue ahead of revealing the anomaly, since the anomaly can at least be detected and dealt with internally 👑.

Now, the opposite case, where the inner and outer carry different ECT values but the outer is ECT(1) and the inner is ECT(0), looks like it needs a similar compromise. But because that case is reversed, no compromise is needed 🦋. It's best to forward the outer whether the transport expects ECT(1) to mean a higher severity than ECT(0) or the same severity. Forwarding the outer either keeps a higher value (if it is higher), or reveals an anomaly to the transport (if the two ECT codepoints mean the same severity) 🌈.

Appendix E. Open Issues 🌷

The new decapsulation behaviour defined in Section 4.2 adds support for propagating two severity levels of congestion 🎉. But transports have no way to discover whether any legacy tunnels on their path won't propagate two severity levels 😬. It would've been nice to add a feature letting transports check path support, but that's still an open issue, and it'll have to be addressed in any future standards action that defines an end-to-end scheme needing two severity levels of congestion. PCN avoids this problem because it only applies to a controlled region, so an operator deploying PCN can upgrade all the legacy tunnels itself 💜.

Author's Address 💌

Bob Briscoe
BT
B54/77, Adastral Park
Martlesham Heath
Ipswich  IP5 3RE
UK

Phone: +44 1473 645196
EMail: bob.briscoe@bt.com
URI:   http://bobbriscoe.net/
📦 where this page comes from