RFC 9049

Path Aware Networking: Obstacles to Deployment (A Bestiary of Roads Not Taken)

📅 published June 2021, from ietf

Path Aware Networking: Obstacles to Deployment ✨

(A Bestiary of Roads Not Taken) 🦋

Title block, bestie, read this first 💅

  • RFC: 9049
  • Stream: Internet Research Task Force (IRTF) 🌈
  • Editor: S. Dawkins, Ed., Tencent America
  • Category: Informational 📖
  • Date: June 2021 📅
  • ISSN: 2070-1721
  • Obsoletes / Updates: this one doesn't list any RFCs it obsoletes or updates 🙅‍♀️

Abstract 🌸

Okay so this doc comes from the Path Aware Networking Research Group (PANRG) 🧚. At their very first meeting, the group decided to catalog and analyze the past attempts to build and deploy Path Aware techniques. Most of those attempts flopped, or only kinda worked 💀. The whole point is to pull out lessons that future Path Aware researchers can actually use 🔥.

So this document is that catalog, plus the analysis 💜.

Status of This Memo 📝

Quick heads up: this is not an Internet Standards Track spec. It's published for information only 🫶.

This is an IRTF product. The IRTF shares the results of its Internet-related research and development, and those results might not be ready for deployment 🧪. This RFC reflects the consensus of the PANRG within the IRTF ✨. Anything the IRSG approves for publication isn't a candidate for any level of Internet Standard, which you can read about in Section 2 of RFC 7841 💅.

For the latest status, errata, and feedback info, check https://www.rfc-editor.org/info/rfc9049 🔗.

Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved 💖.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which are in effect on the publication date. Please actually read them (the link is https://trustee.ietf.org/license-info) because they spell out your rights and restrictions for this document 📜.

Table of Contents 🗂️

   1.  Introduction
     1.1.  What Do "Path" and "Path Awareness" Mean in This Document?
   2.  A Perspective on This Document
     2.1.  Notes for the Reader
     2.2.  A Note about Path Aware Techniques Included in This
           Document
     2.3.  Architectural Guidance
     2.4.  Terminology Used in This Document
     2.5.  Methodology for Contributions
   3.  Applying the Lessons We've Learned
   4.  Summary of Lessons Learned
     4.1.  Justifying Deployment
     4.2.  Providing Benefits for Early Adopters
     4.3.  Providing Benefits during Partial Deployment
     4.4.  Outperforming End-to-End Protocol Mechanisms
     4.5.  Paying for Path Aware Techniques
     4.6.  Impact on Operational Practices
     4.7.  Per-Connection State
     4.8.  Keeping Traffic on Fast Paths
     4.9.  Endpoints Trusting Intermediate Nodes
     4.10. Intermediate Nodes Trusting Endpoints
     4.11. Reacting to Distant Signals
     4.12. Support in Endpoint Protocol Stacks
     4.13. Planning for Failure
   5.  Future Work
   6.  Contributions
     6.1.  Stream Transport (ST, ST2, ST2+)
       6.1.1.  Reasons for Non-deployment
       6.1.2.  Lessons Learned
     6.2.  Integrated Services (IntServ)
       6.2.1.  Reasons for Non-deployment
       6.2.2.  Lessons Learned
     6.3.  Quick-Start TCP
       6.3.1.  Reasons for Non-deployment
       6.3.2.  Lessons Learned
     6.4.  ICMP Source Quench
       6.4.1.  Reasons for Non-deployment
       6.4.2.  Lessons Learned
     6.5.  Triggers for Transport (TRIGTRAN)
       6.5.1.  Reasons for Non-deployment
       6.5.2.  Lessons Learned
     6.6.  Shim6
       6.6.1.  Reasons for Non-deployment
       6.6.2.  Lessons Learned
       6.6.3.  Addendum on Multipath TCP
     6.7.  Next Steps in Signaling (NSIS)
       6.7.1.  Reasons for Non-deployment
       6.7.2.  Lessons Learned
     6.8.  IPv6 Flow Labels
       6.8.1.  Reasons for Non-deployment
       6.8.2.  Lessons Learned
     6.9.  Explicit Congestion Notification (ECN)
       6.9.1.  Reasons for Non-deployment
       6.9.2.  Lessons Learned
   7.  Security Considerations
   8.  IANA Considerations
   9.  Informative References
   Acknowledgments
   Author's Address

1. Introduction 🧚‍♀️

This doc is the group chat for everything IETF folks have learned (the hard way, lol) about Path Aware networking over several decades 😭. It also digs into why a bunch of Path Aware techniques got little or no deployment 💀.

This document represents the consensus of the Path Aware Networking Research Group (PANRG) 💜.

1.1. What Do "Path" and "Path Awareness" Mean in This Document? 🤔

Okay, the first thing reviewers always ask is "What's a Path, and what's Path Awareness?" 🙈 And honestly? Not an easy question to answer here 😅.

Those terms are defined in other PANRG documents [PANRG] and are still being debated in the Research Group as of this writing. But this doc covers work from several decades, so the technologies in Section 6 are way older than the current PANRG definitions of "Path" and "Path Aware." It's very unlikely all the Section 6 contributors shared the same understanding of those words 🫠. Those techs were still called "Path Aware" in early PANRG discussions, so they're in this retrospective anyway 🦋.

Something worth knowing: the definitions in [PANRG-PATH-PROPERTIES] could apply to Path Aware techniques at a bunch of layers of the Internet protocol architecture ([RFC1122], plus several decades of refinements) 🌈. But the contributions for this doc mostly targeted the transport layer and treated a "Path" built by routers as opaque 👀. How well the Lessons Learned here apply at other layers would be a fine topic for follow-on research, so if you're looking for a project, this is it 🍓.

The current PANRG definition of "Path" is in Section 2 ("Terminology") of [PANRG-PATH-PROPERTIES]. I'm including it here for convenience 💖:

   |  Path: A sequence of adjacent path elements over which a packet can
   |  be transmitted, starting and ending with a node.  A path is
   |  unidirectional.  Paths are time-dependent, i.e., the sequence of
   |  path elements over which packets are sent from one node to another
   |  may change.  A path is defined between two nodes.  For multicast
   |  or broadcast, a packet may be sent by one node and received by
   |  multiple nodes.  In this case, the packet is sent over multiple
   |  paths at once, one path for each combination of sending and
   |  receiving node; these paths do not have to be disjoint.  Note that
   |  an entity may have only partial visibility of the path elements
   |  that comprise a path and visibility may change over time.
   |  Different entities may have different visibility of a path and/or
   |  treat path elements at different levels of abstraction.

The current definition of Path Awareness, used by the PANRG, is in Section 1.1 ("Definition") of [PANRG-QUESTIONS]. Also included for convenience ✨:

   |  For purposes of this document, "path aware networking" describes
   |  endpoint discovery of the properties of paths they use for
   |  communication across an internetwork, and endpoint reaction to
   |  these properties that affects routing and/or data transfer.  Note
   |  that this can and already does happen to some extent in the
   |  current Internet architecture; this definition expands current
   |  techniques of path discovery and manipulation to cross
   |  administrative domain boundaries and up to the transport and
   |  application layers at the endpoints.
   |
   |  Expanding on this definition, a "path aware internetwork" is one
   |  in which endpoint discovery of path properties and endpoint
   |  selection of paths used by traffic exchanged by the endpoint are
   |  explicitly supported, regardless of the specific design of the
   |  protocol features which enable this discovery and selection.

2. A Perspective on This Document 🔍

At the first PANRG meeting [PANRG], held at IETF 99 [PANRG-99], Olivier Bonaventure led a talk called "A Decade of Path Awareness" [PATH-Decade] 🎤. It covered the attempts over the past decade to use Path Aware techniques for a bunch of goals, and most of them didn't work out for one reason or another 😩. By the end of that discussion, two things were super clear 💅:

  • The Internet community has built up a LOT of experience with Path Aware techniques over a long time, and
  • Even though some Path Aware techniques did get deployed (Differentiated Services, or Diffserv [RFC2475], is one), most of them never got widespread adoption or deployment 📉. Even "successful" techniques like Diffserv can run into obstacles that stop them from being used more widely 🙃. There are lots of reasons these techniques weren't adopted, or only partly adopted or deployed, and they're definitely worth studying 🔬.

The meta-lessons from all that were:

  • Path Aware networking has been more Research than Engineering, so setting up an IRTF Research Group for it was the right call [RFC7418] 👑.
  • Going through a catalog of past experience to figure out why things weren't adopted would be a great first step for the Research Group 📚.

Allison Mankin, as IRTF Chair, officially chartered the PANRG in July 2018 🥳.

This document is the analysis that group did (Section 4), built on that catalog (Section 6) 🧠.

2.1. Notes for the Reader 📌

This is an Informational doc about Path Aware protocol mechanisms that the Internet Engineering Task Force (IETF) considered, and in some cases standardized 🛠️. It looks at the Lessons Learned from those mechanisms. The goal is to help protocol designers, whether they're in the IRTF, the IETF, or anywhere else in the Internet ecosystem 💜.

Since it's an Informational document published in the IRTF Stream, this one has no authority beyond the quality of its analysis. So it's really just vibes backed by evidence 🧐✨.

2.2. A Note about Path Aware Techniques Included in This Document 🦋

This doc doesn't catalog every proposed Path Aware technique that never got adopted and deployed 🙅‍♀️. Instead, the focus is on technologies that went through the IETF community, and there were still enough of them to give background for the lessons in Section 4, so researchers and protocol engineers can use them 🍓.

No shame, though 🫶. As Section 4 shows, how good a specific technique was had very little to do with whether it got deployed 😤. Based on the techniques cataloged here, it seems likely that the people proposing them were trying to engineer something that couldn't be engineered without doing research first 🧪. The actual shameful move would be not learning from experience, or not sharing that experience with other networking researchers and engineers 💀💅.

2.3. Architectural Guidance ✨

okay so, before we dive into the Lessons Learned, you need to read two IAB docs, bestie 📚🫶 They're "What Makes for a Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption and Subsequent Transitions" [RFC8170]. Total must-reads for getting your head around why some protocols take off and others flop 💀

These two docs aren't specifically about Path Aware networking, but they're still super helpful for anyone who wants to understand what makes a protocol succeed when it gets adopted and deployed. For example, the basic success factors in Section 2.1 of [RFC5218] are a whole vibe for readers of this document 💅

Deployment decisions also come with an economic side, so the IAB Workshop on Internet Technology Adoption and Transition [ITAT] report [RFC7305] is another good one to chew on 🧠💖

Several of the Lessons Learned in Section 4 reflect ideas from [RFC5218], [RFC7305], and [RFC8170]. Iykyk 🌸

2.4. Terminology Used in This Document 🎀

The terms "node" and "element" mean what [PANRG-PATH-PROPERTIES] says they mean in this document. Lock it in 🔒✨

2.5. Methodology for Contributions 🦋

This document grew out of contributions from a bunch of IETF participants who had hands-on experience with one or more Path Aware techniques 🙌

There's so much you could say about the Path Aware techniques people have built. So for this document, contributors were asked to share:

  • the name of a technique, including an abbreviation if one was used.
  • if available, a long-term pointer to the best reference describing the technique.
  • a short description of the problem the technique was intended to solve.
  • a short description of the reasons why the technique wasn't adopted.
  • a short statement of the lessons that researchers can learn from our experience with this technique.

3. Applying the Lessons We've Learned 🍓

The original scope for this document was basically "What mistakes have we made in the decade prior to [PANRG-99], that we shouldn't make again?" 👀 Some of the contributions in Section 6 predate that scope, though. The earliest Path Aware technique mentioned in Section 6 is [IEN-119], which came out in the late 1970s (see Section 6.1). Honestly the networking ecosystem has been evolving nonstop, so it makes total sense to think about how these lessons apply now 🧚

The PANRG looked over the Lessons Learned (Section 4) from the May 23, 2019 draft of this document at IETF 105 [PANRG-105-Min], then kept talking about it at IETF 106 [PANRG-106-Min]. Table 1 shows the "sense of the room" for each lesson after all that discussion. The goal was to figure out whether a lesson is:

  • "Invariant" - well-understood and likely to apply to any proposed Path Aware networking solution.
  • "Variable" - has held back deployment in the past, but might not apply to a specific technique. Doing some engineering analysis to see whether the lesson applies is a smart move 🤓
  • "Not Now" - a characteristic that's basically a minefield full of dragons 🐉💀 Careful network engineers will want to avoid betting on a technique that relies on this, until something big changes.

Section 6.9 on Explicit Congestion Notification (ECN) got added during the review and approval process, after a question from Martin Duke 🙋‍♀️ The March 8, 2021 draft version of Section 6.9 was discussed at [PANRG-110] and is summarized in Section 4.13, which is a brand-new Lesson Learned ✨

    +=====================================================+===========+
    | Lesson                                              | Category  |
    +=====================================================+===========+
    | Justifying Deployment (Section 4.1)                 | Invariant |
    +-----------------------------------------------------+-----------+
    | Providing Benefits for Early Adopters (Section 4.2) | Invariant |
    +-----------------------------------------------------+-----------+
    | Providing Benefits during Partial Deployment        | Invariant |
    | (Section 4.3)                                       |           |
    +-----------------------------------------------------+-----------+
    | Outperforming End-to-End Protocol Mechanisms        | Variable  |
    | (Section 4.4)                                       |           |
    +-----------------------------------------------------+-----------+
    | Paying for Path Aware Techniques (Section 4.5)      | Invariant |
    +-----------------------------------------------------+-----------+
    | Impact on Operational Practices (Section 4.6)       | Invariant |
    +-----------------------------------------------------+-----------+
    | Per-Connection State (Section 4.7)                  | Variable  |
    +-----------------------------------------------------+-----------+
    | Keeping Traffic on Fast Paths (Section 4.8)         | Variable  |
    +-----------------------------------------------------+-----------+
    | Endpoints Trusting Intermediate Nodes (Section 4.9) | Not Now   |
    +-----------------------------------------------------+-----------+
    | Intermediate Nodes Trusting Endpoints               | Not Now   |
    | (Section 4.10)                                      |           |
    +-----------------------------------------------------+-----------+
    | Reacting to Distant Signals (Section 4.11)          | Variable  |
    +-----------------------------------------------------+-----------+
    | Support in Endpoint Protocol Stacks (Section 4.12)  | Variable  |
    +-----------------------------------------------------+-----------+
    | Planning for Failure (Section 4.13)                 | Invariant |
    +-----------------------------------------------------+-----------+

                                  Table 1

"Justifying Deployment", "Providing Benefits for Early Adopters", "Paying for Path Aware Techniques", "Impact on Operational Practices", and "Planning for Failure" were all marked as Invariant. The room agreed these will always be considerations for any proposed Path Aware technique 💯

"Providing Benefits during Partial Deployment" was added after IETF 105, during Research Group Last Call, and it's also Invariant 🌈

For "Outperforming End-to-End Protocol Mechanisms", there's a trade-off 🎢 Path Aware techniques can perform better, but some of them also bring in extra complexity.

  • For example, if you can get the same understanding of path characteristics from measurements taken over a few more round trips, endpoint implementers probably won't be rushing to add complexity. Plus, lots of attributes can be measured from an endpoint without any help from intermediate nodes 🙈

For "Per-Connection State", the Research Group's big questions were "how much state" and "where state is maintained" 🤔

  • Integrated Services (IntServ) (Section 6.2) needed state at every participating intermediate node, for every connection between two endpoints. 😩 But as the Internet has evolved, it's become more common to carry many connections inside a tunnel that looks like a single connection to intermediate nodes. So extra end-to-end connections don't add extra state to the intermediate nodes between tunnel endpoints. If those tunnels are encrypted, the intermediate nodes between tunnel endpoints can't tell connections apart, even if they wanted to 🔐

For "Keeping Traffic on Fast Paths", we noticed this was true for many platforms, but not all of them 🏎️

  • For backbone routers, this is likely an Invariant. But for platforms that rely more on general-purpose computers to make forwarding decisions, it may not be a fatal flaw for Path Aware techniques 💜

For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes Trusting Endpoints", these lessons point to a broader need to revisit the Internet Threat Model 🛡️

  • We were relieved to see that discussions about this were already happening in the IETF community at IETF 105. (See the Security Area Open Meeting minutes [SAAG-105-Min] for the discussion of [INTERNET-THREAT-MODEL] and [FARRELL-ETM].) The Internet Architecture Board also set up a mailing list for continued discussions [model-t]. Still, there are Path Aware networking aspects of this work that need research 🔬

For "Reacting to Distant Signals", we noticed that not all attributes are equal 📡

  • If an attribute stays stable over an extended period of time, is hard to observe through end-to-end mechanisms, and is valuable, then Path Aware techniques that rely on that attribute to provide a significant benefit become more attractive 🌟
  • Analysis that helps identify attributes useful enough to justify deploying Path Aware techniques that use them would be super helpful 🙌

For "Support in Endpoint Protocol Stacks", we noticed that Path Aware applications must be able to identify and communicate requirements about path characteristics 💅

  • The de facto sockets API has no way of signaling application expectations for the network path to the protocol stack. Big oof 😭

4. Summary of Lessons Learned ✨

This section sums up the Lessons Learned from the contributed subsections in Section 6 🧾

Each Lesson Learned is tagged with one or more contributions that ran into this obstacle as a significant impediment to deployment. Other contributed techniques may have hit the same obstacle too, but it might not have been the biggest blocker for them 🫶

Sometimes an obstacle impedes deployment, and other times that same obstacle prevents adoption and deployment entirely 🥲 The Research Group talked about separating obstacles that impede from obstacles that prevent, but the line between "impede" and "prevent" seems to move over time. Some of the Lessons Learned are based on both a) Path Aware techniques that never got deployed and b) Path Aware techniques that did get deployed, but not widely or quickly. For examples of this shifting line, see Sections 6.6 and 6.6.3 👀

4.1. Justifying Deployment 💰

The benefit of Path Awareness has to be big enough to justify making changes in an operational network. The colloquial U.S. American English expression, "If it ain't broke, don't fix it", is a "best current practice" on today's Internet 😂 (See Sections 6.3, 6.4, 6.5, and 6.9, in addition to [RFC5218].)

4.2. Providing Benefits for Early Adopters 🚀

Providing benefits for early adopters can be key 🔑 If everyone has to deploy a technique before it provides any benefit, or even before it works at all, the technique is unlikely to be adopted widely or quickly 😩 (See Sections 6.2 and 6.3, in addition to [RFC5218].)

4.3. Providing Benefits during Partial Deployment ✨

Okay so, some proposals only pay off once EVERY single path element from one end to the other has been upgraded to support the new technique 😭. That's a LOT of coordination between operators who each control a slice of the path, and between operators and end users if the endpoints need upgrades too. 💀 If a technique still gives benefits when only part of the path is upgraded, bestie, that's giving "adoption is coming" energy 🦋. (See Sections 6.2, 6.3, and 6.9, in addition to [RFC5218].)

4.4. Outperforming End-to-End Protocol Mechanisms ✨

Adaptive end-to-end protocol mechanisms can respond to feedback super fast 🔥. So the extra benefit a new Path Aware mechanism gets from poking at nodes along a path, or peeking at their attributes, might be WAY smaller than everyone expected 🙈. (See Sections 6.3 and 6.5.)

4.5. Paying for Path Aware Techniques ✨

"Follow the money." 💅 If operators can't charge for a Path Aware technique to recover what it cost to deploy, then the benefits to the operator have to be really significant 👑. Corollary: if operators DO charge for it, the benefits to users have to be significant enough to justify the price 🫶. (See Sections 6.1, 6.2, 6.5, and 6.9.)

4.6. Impact on Operational Practices ✨

When a Path Aware technique needs changes to operational practices, that can totally affect how fast or how widely a promising technique gets deployed 🌸. Those changes can make deployment more likely, but ngl they often discourage it too 😬. (See Section 6.6, including Section 6.6.3.)

4.7. Per-Connection State ✨

Per-connection state in intermediate nodes has been an impediment to adoption and deployment in the past, because it adds cost and complexity 💀. Often you can get similar benefits with much less finely grained state 🧚. This is especially true as you move from the edge of the network further into the routing core 🌈. (See Sections 6.1 and 6.2.)

4.8. Keeping Traffic on Fast Paths ✨

Lots of modern platforms, especially high-end routers, have hardware that's amazing at simple per-packet forwarding decisions ("fast paths") 🔥. But they weren't really designed to make heavy use of in-band mechanisms like IPv4 and IPv6 Router Alert Options (RAOs), which need more processing to make forwarding decisions 😭. Packets carrying in-band mechanisms get diverted to other processors in the router, and those run at much lower packet-processing rates 💅. So operators can be reluctant to deploy techniques that lean heavily on in-band mechanisms, because they may significantly reduce packet throughput 🙈. (See Section 6.7.)

4.9. Endpoints Trusting Intermediate Nodes ✨

If intermediate nodes along the path can't be trusted, it's unlikely that endpoints will rely on signals from those nodes to drive changes to endpoint behavior 🚫. BUT "trust" is not binary, bestie 💖! One low level of trust applies when a node receiving a message can confirm that the sender has visibility of the packets on the path it's trying to control [RFC8085] (for example, an ICMP Destination Unreachable message [RFC0792] that includes the Internet Header + 64 bits of Original Data Datagram payload from the source 🧐). A higher level of trust can arise when an endpoint has set up a short-term, or even long-term, trust relationship with network nodes 🫶. (See Sections 6.4 and 6.5.)

4.10. Intermediate Nodes Trusting Endpoints ✨

If the endpoints don't have any trust relationship with the intermediate nodes along a path, operators have been reluctant to deploy techniques that rely on endpoints sending unauthenticated control signals to routers 😤. (See Sections 6.2 and 6.7.) (Also, this is still a factor holding back Diffserv deployment 🥲.)

4.11. Reacting to Distant Signals ✨

The Internet is a distributed system, so if information from distant path elements has to travel really far to reach a Path Aware host, it might not accurately reflect the state of the distant host or the path elements by the time it arrives locally 🌪️. When that happens, the benefit a Path Aware technique provides can be inconsistent and won't always be helpful 😭. (See Section 6.3.)

4.12. Support in Endpoint Protocol Stacks ✨

Just because a protocol stack has a new feature or signal doesn't mean applications will actually use it 🙃. Protocol stacks might not know how to use Path Aware techniques well, since the stack may need information from applications to make the technique work, and applications may not know that information a priori 🤷‍♀️. Even when an application does know it, the de facto sockets API has no way to signal those path expectations down to the protocol stack 😩. So if applications are going to give these expectations to protocol stacks, we need an API that signals more than just the packets to be sent 🎀. (See Sections 6.1 and 6.2.)

4.13. Planning for Failure ✨

If early implementers find severe problems with a new feature, that feature will likely get disabled, and getting implementers to re-enable it can be SO hard, like years or even decades 💀. Testing helps, and so do partial deployments for a subset of users 🧪, instrumentation that detects degraded user experience 📊, and even "failback" to a previous version or "failover" to an entirely different implementation 🔄. (See Section 6.9.)

  1. Future Work ✨

Okay, so this document has been pretty retrospective 🕰️. Besides thinking about how the Lessons Learned so far apply to current and future Path Aware networking proposals, it's also worth asking whether there's deeper investigation left to do 🔍.

  • We note that this work was based on contributions from experts on various Path Aware techniques, and all of the contributed techniques involved unicast protocols 📡. We didn't consider how these lessons might apply to multicast. But given anecdotal reports at the IETF 109 Media Operations (MOPS) Working Group meeting of IP multicast offerings within data centers at one or more cloud providers [MOPS-109-Min], it might be useful to think about Path Awareness in multicast before we have a history of unsuccessful deployments to document 📝.
  • The question of whether a mechanism supports admission control, based on either endpoints or applications, is tied to Path Awareness 🎟️. One of the motivations of IntServ and a number of other architectures (for example, Deterministic Networking [RFC8655]) is the ability to "say no" to an application based on resource availability on a path. That happens before the application injects traffic onto the path and discovers the path can't sustain enough utility to meet its minimum needs 🚦. Whether admission control is needed keeps coming up, but we've learned a few useful lessons. They're covered implicitly in some of the Lessons Learned in this document, but they might be spelled out explicitly here 💡:
    • We have gained a lot of experience with application-based adaptation since the days when applications just injected traffic inelastically into the network 📈. Those adaptations seem to work well enough that admission control is worth less to these applications 🙌.
    • There are end-to-end measurement techniques that can steer traffic at the application layer, like Content Delivery Networks (CDNs) and multi-CDNs such as Conviva [Conviva], etc. 🌐
    • We noted in Section 4.12 that applications often don't know how to use Path Aware techniques 🥺. That includes not knowing enough about their admission control threshold to ask accurately for the resources they need. Whether that's because the application itself doesn't know, or because it has no way to signal its expectations to the underlying protocol stack, attempts to help them haven't gotten anywhere so far. For example, the multiple-TSPEC (Traffic Specification) additions to RSVP, which tried to mirror codec selection by applications [INTSERV-MULTIPLE-TSPEC], expired in 2013 🥀.
  • We note that this work took the then-current IP network architecture as given, at least at the time each technique was proposed 🏛️. It might be useful to consider which aspects of the now-current IP network architecture ease, or impede, Path Aware techniques 🧐. For example, IP offers limited ability to constrain bidirectional paths to be symmetric, and information-centric networking protocols such as Named Data Networking (NDN) and Content-Centric Networking (CCNx) [RFC8793] must force bidirectional path symmetry using protocol-specific mechanisms 🔐.
  1. Contributions ✨

Contributions on these Path Aware techniques were analyzed to arrive at the Lessons Learned captured in Section 4 📚.

Most readers won't need to read through this section carefully 🫠. But we wanted to record these hard-fought lessons as a service to anyone who revisits this document, so the details are right there at hand 🎁.

6.1. Stream Transport (ST, ST2, ST2+) 🌊

The suggested references for Stream Transport are:

  • "ST - A Proposed Internet Stream Protocol" [IEN-119] 📜
  • "Experimental Internet Stream Protocol: Version 2 (ST-II)" [RFC1190] 📜
  • "Internet Stream Protocol Version 2 (ST2) Protocol Specification - Version ST2+" [RFC1819] 📜

The first version of Stream Transport, ST [IEN-119], was published in the late 1970s and was implemented and deployed on the ARPANET at small scale 🕹️. It was used throughout the 1980s for experimental transmission of voice, video, and distributed simulation 🎙️🎥.

The second version of the ST specification (ST2) [RFC1190] [RFC1819] was an experimental connection-oriented internetworking protocol that operated at the same layer as connectionless IP 🔗. ST2 packets could be told apart by their IP header version numbers: IP used version number 4 at that time, while ST2 used version number 5 🔢.

ST2 used a control plane layered over IP to select routes and reserve capacity for real-time streams across a network path. This was based on a flow specification communicated by a separate protocol 🛤️. The flow specification could be tied to QoS state in routers, producing an experimental resource reservation protocol 🎟️. This allowed ST2 routers along a path to offer end-to-end guarantees, mainly to meet the QoS requirements for real-time services over the Internet 📞✨.

6.1.1. Reasons for Non-deployment 🙈

Even though it was implemented in a range of equipment, ST2 wasn't widely used after the experiments wrapped up 🥲. It didn't offer the scalability and fate-sharing properties that the Internet community has come to want 📉.

The ST2 protocol is no longer in use 🪦.

6.1.2. Lessons Learned

okay so as time went on, the trade-off between router processing and link capacity totally shifted 💅 Links got faster, and router processing got comparatively more expensive 😭

The ST2 control protocol used "hard state" 🧱. Once a route was established and resources were reserved, the routes and resources just stayed there until they were explicitly released via signaling. A soft-state approach was thought to be superior to this hard-state approach, and that thinking led to the development of the IntServ model described in Section 6.2 ✨

6.2. Integrated Services (IntServ)

the suggested references for IntServ are, bestie 📚:

  • "Integrated Services in the Internet Architecture: an Overview" [RFC1633]
  • "Specification of the Controlled-Load Network Element Service" [RFC2211]
  • "Specification of Guaranteed Quality of Service" [RFC2212]
  • "General Characterization Parameters for Integrated Service Network Elements" [RFC2215]
  • "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification" [RFC2205]

In 1994, when the IntServ architecture document [RFC1633] came out, real-time traffic was just starting to show up on the Internet 🎀 Bandwidth was still a scarce commodity back then. Internet Service Providers built their networks over DS3 (45 Mbps) infrastructure, and sub-rate (< 1 Mbps) access was common. So the IETF saw a need for a fine-grained QoS mechanism, and honestly that makes sense 🌸

In the IntServ architecture, some applications can require service guarantees. So those applications use RSVP [RFC2205] to signal QoS reservations across network paths. Every router in the network that takes part in IntServ keeps per-flow soft state to a) perform call admission control and b) deliver guaranteed service 🧚

Applications use Flow Specifications (Flow Specs, or FLOWSPECs) [RFC2210] to describe the traffic that they emit. RSVP reserves capacity for traffic on a per-Flow-Spec basis 💜

6.2.1. Reasons for Non-deployment

ngl this is the part where IntServ hits a wall 🧱 Although IntServ has been used in enterprise and government networks, it was never widely deployed on the Internet because of its cost. The following factors contributed to the operational cost:

  • IntServ must be deployed on every router that is on a path where IntServ is to be used. Although it is possible to include a router that does not participate in IntServ along the path being controlled, if that router is likely to become a bottleneck, IntServ cannot be used to avoid that bottleneck along the path.
  • IntServ maintained per-flow state.

As IntServ was being discussed, the following happened 👀

  • For many expected uses, it became more cost effective to solve the QoS problem by adding bandwidth. Between 1994 and 2000, Internet Service Providers upgraded their infrastructures from DS3 (45 Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint was using IntServ in an IntServ-enabled network, its requests would rarely, if ever, be denied. So endpoints and Internet Service Providers had little reason to enable IntServ 🙈
  • Diffserv [RFC2475] offered a more cost-effective, albeit less fine-grained, solution to the QoS problem 🍓

6.2.2. Lessons Learned

the lessons from this era are 🫶

  • Any mechanism that requires every participating on-path router to maintain per-flow state is not likely to succeed, unless the additional cost for offering the feature can be recovered from the user.
  • Any mechanism that requires an operator to upgrade all of its routers is not likely to succeed, unless the additional cost for offering the feature can be recovered from the user.

In environments where IntServ has been deployed, trust relationships with endpoints are very different from trust relationships on the Internet itself. There are often clearly defined hierarchies in Service Level Agreements (SLAs) governing well-defined transport flows, operating with predetermined capacity and latency requirements over paths where capacity or other attributes are constrained. That's a whole different vibe from the open Internet 🌈

IntServ was never widely deployed to manage capacity across the Internet. However, the technique that it produced was deployed for reasons other than bandwidth management, and that's the iconic plot twist ✨ RSVP is widely deployed as an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter Specs to distribute firewall filters, although they are called "Flow Spec Component Types" in BGP [RFC5575] 🔥

6.3. Quick-Start TCP

the suggested references for Quick-Start TCP are 📎

  • "Quick-Start for TCP and IP" [RFC4782]
  • "Determining an appropriate sending rate over an underutilized network path" [SAF07]
  • "Fast Startup Internet Congestion Control for Broadband Interactive Applications" [Sch11]
  • "Using Quick-Start to enhance TCP-friendly rate control performance in bidirectional satellite networks" [QS-SAT]

Quick-Start is defined in an Experimental RFC [RFC4782] 🧪 It's a TCP extension that leverages support from the routers on the path to determine an allowed initial sending rate for a path through the Internet, either at the start of data transfers or after idle periods. Without information about the path, a sender can't easily figure out an appropriate initial sending rate. So the default TCP congestion control uses the safe but time-consuming slow-start algorithm [RFC5681] 🐢 With Quick-Start, connections are allowed to use higher initial sending rates if there is significant unused bandwidth along the path, and if the sender and all of the routers along the path approve the request ✨

By examining the Time To Live (TTL) field in Quick-Start packets, a sender can determine if routers on the path have approved the Quick-Start request. However, this method is unable to take into account the routers hidden by tunnels or other network nodes that are invisible at the IP layer 🙈

The protocol also includes a nonce that provides protection against cheating routers and receivers 🔐 If the Quick-Start request is explicitly approved by all routers along the path, the TCP host can send at up to the approved rate. Otherwise, TCP would use the default congestion control. Quick-Start requires modifications in the involved end systems as well as in routers. Due to the resulting deployment challenges, Quick-Start was only proposed in [RFC4782] for controlled environments 💅

The Quick-Start mechanism is a lightweight, coarse-grained, in-band, network-assisted fast startup mechanism 🦋 The benefits were studied by simulation in a research paper [SAF07] that complements the protocol specification. The study confirms that Quick-Start can significantly speed up mid-sized data transfers. That paper also presents router algorithms that do not require keeping per-flow state 👑 Later studies [Sch11] comprehensively analyze Quick-Start with a full Linux implementation and with a router fast-path prototype using a network processor. In both cases, Quick-Start could be implemented with limited additional complexity.

6.3.1. Reasons for Non-deployment

okay but here's the tea ☕ Experiments with Quick-Start in [Sch11] revealed several challenges:

  • Having information from the routers along the path can reduce the risk of congestion, but it cannot avoid it entirely. Determining whether there is unused capacity is not trivial in actual router and host implementations. Data about available capacity visible at the IP layer may be imprecise, and due to the propagation delay, information can already be outdated when it reaches a sender. There is a trade-off between the speedup of data transfers and the risk of congestion, even with Quick-Start. This could be mitigated by only allowing Quick-Start to access a proportion of the unused capacity along a path.
  • For scalable router fast-path implementations, it is important to enable parallel processing of packets, as this is a widely used method, e.g., in network processors. One challenge is synchronization of information between packets that are processed in parallel, which should be avoided as much as possible 🧩
  • Only some types of application traffic can benefit from Quick-Start. Capacity needs to be requested and discovered. The discovered capacity needs to be utilized by the flow, or it implicitly becomes available for other flows. Failing to use the requested capacity may have already reduced the pool of Quick-Start capacity that was made available to other competing Quick-Start requests. The benefit is greatest when senders use this only for bulk flows and avoid sending unnecessary Quick-Start requests, e.g., for flows that only send a small amount of data. Choosing an appropriate request size requires application-internal knowledge that is not commonly expressed by the transport API. How a sender can determine the rate for an initial Quick-Start request is still a largely unsolved problem 🤯

There is no known deployment of Quick-Start for TCP or other IETF transports 🥲

6.3.2. Lessons Learned

some lessons can be learned from Quick-Start 📝 Despite being a very lightweight protocol, Quick-Start suffers from poor incremental deployment properties, both a) regarding the required modifications in network infrastructure and b) regarding its interactions with applications. Except for corner cases, congestion control can be quite efficiently performed end to end in the Internet, and in modern stacks there is not much room for significant improvement from additional network support 💖

After the Quick-Start specification was published, there were large-scale experiments with an initial window of up to 10 segments [RFC6928]. This alternative "IW10" approach can also ramp up data transfers faster than standard congestion control, but it only requires sender-side modifications. As a result, this approach can be easier to deploy incrementally in the Internet 🚀 While theoretically Quick-Start can outperform "IW10", the improvement in data transfer completion times can, in many cases, be small. After [RFC6928] was published, most modern TCP stacks increased their default initial window 🎉

6.4. ICMP Source Quench

the suggested reference for ICMP Source Quench is 🛰️

  • "Internet Control Message Protocol" [RFC0792]

The ICMP Source Quench message [RFC0792] allowed an on-path router to request that the source of a flow reduce its sending rate. This method let a router give an early heads-up of impending congestion on a path to the sources contributing to that congestion 🌸

6.4.1. Reasons for Non-deployment

okay so this method actually got deployed in Internet routers for a while 🌸 but endpoints had super mixed reactions to getting it 🙈 For low-speed links with not much multiplexing of flows, it could be used to regulate (momentarily reduce) the transmission rate. BUT the simple signal does not scale with link speed or with the number of flows sharing a link 💀

The approach got overtaken by the evolution of congestion control methods in TCP [RFC2001], and later by other IETF transports too 🦋 These newer methods were based on measuring the end-to-end path with an algorithm in the endpoint, so they could evolve and mature way faster than methods that relied on interactions between operational routers and endpoint stacks. literally the glow-up was inevitable ✨

After ICMP Source Quench was specified, the IETF started recommending that transports provide end-to-end congestion control [RFC2001]. The Source Quench method has been obsoleted by the IETF [RFC6633], and both hosts and routers must now silently discard this message. bye bye, Source Quench 👋💜

6.4.2. Lessons Learned

this method had several problems, bestie 😭

First, [RFC0792] did not sufficiently specify how the sender would react to the ICMP Source Quench signal from the path (e.g., [RFC1016]). There was ambiguity in how the sender should use this extra information. This could lead to unfairness in the way that receivers (or routers) responded to this message. not the vibe 🙅‍♀️

Second, while the message did provide additional information, the Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a more robust and informative signal for network nodes to give early indication that a path has become congested. ECN really said "hi, I'm the upgrade" 💅🌈

The mechanism originated at a time when the Internet trust model was very different 🕰️ Most endpoint implementations did not try to verify that the message came from an on-path node before using it. This made it vulnerable to Denial-of-Service (DoS) attacks. In theory, routers might have chosen to use the quoted packet in the ICMP payload to validate that the message came from an on-path node, but this would have increased per-packet processing overhead for each router along the path. It also would have required transport functionality in the router to verify whether the quoted packet header matched a packet the router had sent. Ugh, so much work. 🙈 On top of that, Section 5.2 of [RFC4443] noted ICMPv6-based attacks on hosts that would also have threatened routers processing ICMPv6 Source Quench payloads. As time passed, it became increasingly obvious that the lack of validation exposed receivers to a security vulnerability: the messages could be forged to create a tangible DoS opportunity. 🚨 this is the part where the villain arc starts 💀

6.5. Triggers for Transport (TRIGTRAN) 🦋

The suggested references for TRIGTRAN are:

  • TRIGTRAN BOF at IETF 55 [TRIGTRAN-55]
  • TRIGTRAN BOF at IETF 56 [TRIGTRAN-56]

okay so TCP [RFC0793] has a well-known weakness 😩 Its end-to-end flow control mechanism has only a single signal: the loss of a segment, detected when no acknowledgment for the lost segment is received at the sender. There are lots of reasons the sender might not get an acknowledgment for a segment. The segment could have been trapped in a routing loop, damaged in transmission and failed checksum verification at the receiver, or lost because some intermediate device discarded the packet. Or any of a variety of other things could have happened to the acknowledgment on the way back from the receiver to the sender. TCP implementations since the late 1980s made the "safe" decision and interpreted the loss of a segment as evidence that the path between two endpoints may have become congested enough to exhaust buffers on intermediate hops. So the TCP sender should "back off," reducing its sending rate until it knows its segments are being delivered without loss [RFC5681]. cautious, but kinda dramatic 🙃

The thinking behind TRIGTRAN was this: if a path completely stopped working because a link along it was "down," something along the path could signal TCP when that link returned to service. The sending TCP could then retry immediately, without waiting for a full retransmission timeout (RTO) period. the dream was giving us main character energy ⚡🎀

6.5.1. Reasons for Non-deployment

the early dreams for TRIGTRAN got dashed because of an assumption that TRIGTRAN triggers would be unauthenticated 💔 This meant that any "safe" TRIGTRAN mechanism would have needed something like setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing that it was 254 upon receipt. That way, a receiver could verify that a signal came from an adjacent sender known to be on the path being used, and not some unknown sender that might not even be on the path (e.g., "The Generalized TTL Security Mechanism (GTSM)" [RFC5082]). This situation is very similar to the case for ICMP Source Quench messages described in Section 6.4. Those were also unauthenticated and could be sent by an off-path attacker, which resulted in deprecation of ICMP Source Quench message processing [RFC6633]. the same story, different protocol 🥲

TRIGTRAN's scope shrunk from "the path is down" to "the first-hop link is down." 📉

But things got worse. 😩

Because TRIGTRAN triggers would only be provided when the first-hop link was "down," they couldn't replace normal TCP retransmission behavior if the path failed because some link further along the network path was "down." So TRIGTRAN triggers added complexity to an already-complex TCP state machine, and they did not allow any existing complexity to be removed. the audacity 💀

There was also an issue that the TRIGTRAN signal was not sent in response to a specific host that had been sending packets. Instead it was a signal that stimulated a response from any sender on the link. This needs to scale when there are multiple flows trying to use the same resource, yet the sender of a trigger has no idea how many of the potential traffic sources will respond by sending packets. If recipients of the signal "back off" their responses to a trigger to improve scaling, then that immediately mitigates the benefit of the signal. so it's giving self-sabotage 🙃

Finally, intermediate forwarding nodes required modification to provide TRIGTRAN triggers. But operators couldn't charge for TRIGTRAN triggers, so there was no way to recover the cost of modifying, testing, and deploying updated intermediate nodes. no money, no upgrade, no cap 💸

Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56 [TRIGTRAN-56]. But this work was not chartered, and there was no interest in deploying TRIGTRAN unless it was chartered and standardized in the IETF. 🥀

6.5.2. Lessons Learned

the reasons this work was not chartered, much less deployed, give us some really useful lessons for researchers 📚🫶

  • TRIGTRAN started with a plausible value proposition, but networking realities in the early 2000s forced reductions in scope. Those reductions led directly to reductions in potential benefits, but with no corresponding reductions in costs and complexity. 💅
  • These reductions in scope were the direct result of an inability for hosts to trust or authenticate TRIGTRAN signals they received from the network. 🔐
  • Operators did not believe they could charge for TRIGTRAN signaling, because first-hop links didn't fail frequently and TRIGTRAN provided no reduction in operating expenses. So there was little incentive to purchase and deploy TRIGTRAN-capable network equipment. 💸🙈

It is also worth noting that the targeted environment for TRIGTRAN in the late 1990s contained links with a relatively small number of directly connected hosts, for instance cellular or satellite links. The transport community was well aware of the dangers of sender synchronization, where multiple senders receive the same stimulus at the same time. But the working assumption for TRIGTRAN was that there wouldn't be enough senders for this to be a meaningful problem. In the 2010s, it was common for a single "link" to support many senders and receivers. That likely would have required TRIGTRAN senders to wait some random amount of time before sending after receiving a TRIGTRAN signal, which would have reduced the benefits of TRIGTRAN even more. the timeline really said "not this one" 🕰️💀

6.6. Shim6 🧚

The suggested reference for Shim6 is:

  • "Shim6: Level 3 Multihoming Shim Protocol for IPv6" [RFC5533]

okay so the IPv6 routing architecture [RFC1887] assumed that most sites on the Internet would be identified by Provider Assigned IPv6 prefixes. That meant Default-Free Zone routers only contained routes to other providers, which resulted in a very small IPv6 global routing table. tiny table, big vibes 🌸

For a single-homed site, this could work really well. A multihomed site with only one upstream provider could also work well. Still, BGP multihoming from a single upstream provider was often a premium service (costing more than twice as much as two single-homed sites). And if that single upstream provider went out of service, all of the multihomed paths could fail simultaneously. 😬

IPv4 sites often multihomed by obtaining Provider Independent prefixes and advertising these prefixes through multiple upstream providers. With the assumption that any multihomed IPv4 site would also multihome in IPv6, it seemed likely that IPv6 routing would face the same pressures to announce Provider Independent prefixes. That would result in an IPv6 global routing table with the same explosive growth as the IPv4 global routing table. 📈💀 During the early 2000s, work began on a protocol that would provide multihoming for IPv6 sites without requiring sites to advertise Provider Independent prefixes into the IPv6 global routing table. the solution arc has entered the chat 🦋

This protocol, called "Shim6," allowed two endpoints to exchange multiple addresses ("Locators") that all mapped to the same endpoint ("Identity"). After an endpoint learned multiple Locators for the other endpoint, it could send to any of those Locators with the expectation that the packets would all be delivered to the endpoint with the same Identity. Shim6 was an example of an "Identity/Locator Split" protocol. ✨

Shim6, as defined in [RFC5533] and related RFCs, provided a workable solution for IPv6 multihoming using Provider Assigned prefixes. It included capability discovery and negotiation, and it allowed end-to-end application communication to continue even in the face of path failure. That's because applications don't see Locator failures and keep communicating with the same Identity using a different Locator. literally iconic 👑💖

6.6.1. Reasons for Non-deployment ✨

okay so bestie, the problem Shim6 was supposed to solve was "site multihoming" 🏠, but what it actually gave you was "host multihoming" 💻. That meant the call on which path to use was made by the host, not by the edge router. Main character energy, but for the wrong character 🙈

Shim6 could have gotten a better technical solution with more work, ngl 🫶. But the biggest blockers to deploying it were operational and business stuff 💀. Those got talked about at multiple network operator group meetings, including [Shim6-35] at [NANOG-35] 🎀.

On the technical side, people were worried that Shim6 made the host track all the connections while also tracking Identity/Locator mappings in the kernel, and tracking failures so it could notice when an available path had died 😭.

The operational worries were about traffic engineering. Operators were steering traffic in aggregates, and with Shim6 those traffic engineering policies must be pushed down to individual hosts 🌸. Operators also wouldn't have any visibility or control over a host deciding to switch to another path 👀. They were concerned that letting hosts steer traffic could make operator networks oscillate because of feedback loops, if hosts kept hopping between paths a lot 🌀. And since Shim6 was meant to support multihoming across operators, any operator carrying only one of the paths would have even less visibility, with traffic suddenly appearing and vanishing on their network 🫠.

Firewalls were another problem 🧱. A firewall expecting to find a TCP or UDP transport-level protocol header in the IP payload would instead see a Shim6 Identity header. Its normal processing logic wouldn't look past that header, so it couldn't do its transport-protocol-based firewalling. It would fall back to its default action, which would most likely be to drop any packet that doesn't match a processing rule 💀.

The business issues came down to operators not wanting to reduce or remove their ability to sell BGP multihoming service to their own customers 💅. That service is often more expensive than two single-homed connectivity services, so it's a whole vibe to protect 💜.

6.6.2. Lessons Learned

It is extremely important to take operational concerns into account when a Path Aware protocol is making decisions about path selection that may conflict with existing operational practices and business considerations 🔥. Bestie, you cannot just ignore the people who run the network 😭✨.

6.6.3. Addendum on Multipath TCP

During discussions in the PANRG session at IETF 103 [PANRG-103-Min] 🌈, Lars Eggert, past Transport Area Director, pointed out something juicy. During the charter discussions for the Multipath TCP Working Group [MP-TCP], operators worried that customers could use Multipath TCP to load-share TCP connections across operators at the same time, and compare passive performance measurements across network paths in real time 👀. That would shift the balance of power in those business relationships 💀. The Multipath TCP Working Group still got chartered, but this concern could have turned into an obstacle to deployment anyway 🦋.

Operator objections to Shim6 were focused on technical concerns. But this one could also have become an obstacle to Shim6 deployment, if the technical concerns had been overcome 🎀.

6.7. Next Steps in Signaling (NSIS) 🧚

The suggested references for Next Steps in Signaling (NSIS) are:

  • the concluded working group charter [NSIS-CHARTER-2001]
  • "GIST: General Internet Signalling Transport" [RFC5971]
  • "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)" [RFC5973]
  • "NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service Signaling" [RFC5974]
  • "Authorization for NSIS Signaling Layer Protocols" [RFC5981]

The NSIS Working Group worked on signaling techniques for network-layer resources, like QoS resource reservations and Firewall and NAT traversal 🌸.

When RSVP [RFC2205] was being used in deployments, a bunch of questions popped up about its perceived limitations and potential missing features 🤔. The NSIS Working Group charter [NSIS-CHARTER-2001] listed issues like interworking between domains with different QoS architectures, mobility and roaming for IP interfaces, and complexity. Later on, people also realized RSVP had no security [RFC4094] 😭.

NSIS was chartered to tackle all of that. It started out focused on QoS signaling as its main use case. Over time, though, a new approach evolved with a modular architecture using two application-specific signaling protocols: a) the NSIS Signaling Layer Protocol (NSLP), sitting on top of b) a generic signaling transport protocol, the NSIS Transport Layer Protocol (NTLP) ✨.

NTLP is defined in [RFC5971]. Two types of NSLPs are defined: one for QoS signaling [RFC5974] and one for NATs/firewalls [RFC5973] 🎀.

6.7.1. Reasons for Non-deployment

The obstacles to deployment can be grouped into implementation-related aspects and operational aspects 🔍.

  • Implementation-related aspects:

NSIS does provide benefits for flexibility, mobility, and security compared to other network signaling techniques. But hardware vendors were reluctant to deploy it, because it would need extra implementation effort and would add complexity to router implementations 🙄.

NTLP mainly operates as a path-coupled signaling protocol. That means its messages get processed at the control plane of each intermediate node that is also forwarding the data flows 🌀. This needs a mechanism to intercept signaling packets while they're forwarded in the same way as data packets, especially along the same path. NSIS uses the IPv4 and IPv6 Router Alert Option (RAO) to intercept those path-coupled signaling messages. This technique requires router implementations to correctly understand and handle RAOs, for example by only processing packets with RAOs of interest and leaving packets with irrelevant RAOs in the fast forwarding processing path (a comprehensive discussion of these issues can be found in [RFC6398]). The latter was an issue with some router implementations at the time of standardization 💀.

Another reason is that path-coupled signaling protocols, which interact with routers and ask them to manipulate state (or any other network element in general), are under scrutiny 👀. A packet, or a sequence of packets, coming from the mainly untrusted data path is requesting the creation and manipulation of network state. That's seen as potentially dangerous, for example it opens up a DoS threat to a router's control plane, and it's hard for an operator to control 🚨. Path-coupled signaling approaches were considered problematic (see also Section 3 of [RFC6398]). There are recommendations on how to secure NSIS nodes and deployments (for example [RFC5981]) 🔐.

  • Operational Aspects:

NSIS didn't just need trust between customers and their provider. It also needed trust among different providers 🤝. QoS signaling techniques would need some kind of dynamic SLA support, which would imply (potentially quite complex) bilateral negotiations between different Internet Service Providers. That complexity wasn't considered worth it. Increasing the bandwidth, and thus avoiding bottlenecks, was cheaper than actively managing network resource bottlenecks with path-coupled QoS signaling 💸. On top of that, an end-to-end path typically involves several provider domains, and those providers need to cooperate closely when something fails 😩.

6.7.2. Lessons Learned

One goal of NSIS was to decrease the complexity of the signaling protocol. But a path-coupled signaling protocol comes with the intrinsic complexity of IP-based networks, on top of the complexity of the signaling protocol itself 🌪️. Sources of that intrinsic complexity include:

  • the presence of asymmetric routes between endpoints and routers.
  • the lack of security and trust at large in the Internet infrastructure.
  • the presence of different trust boundaries.
  • the effects of best-effort networks (for example, robustness to packet loss).
  • divergence from the fate-sharing principle (for example, state within the network).

Any path-coupled signaling protocol has to deal with these realities 💅.

Operators look at the use of IPv4 and IPv6 Router Alert Options (RAOs) to signal routers along the path from end systems with suspicion 🙃. Those end systems usually aren't authenticated, and heavy use of RAOs can easily increase the CPU load on routers that are built to process most packets through a hardware "fast path" and divert packets carrying RAOs to a slower, more capable processor 🐢.

6.8. IPv6 Flow Labels 🌈

The suggested reference for IPv6 Flow Labels is:

  • "IPv6 Flow Label Specification" [RFC6437]

IPv6 specifies a 20-bit Flow Label field [RFC6437] 🍓. It sits in the fixed part of the IPv6 header, so it's present in every IPv6 packet. An endpoint sets the value in this field to one of a set of pseudorandomly assigned values. If a packet isn't part of any flow, the flow label value is set to zero [RFC3697]. A number of Standards Track and Best Current Practice RFCs (for example [RFC8085], [RFC6437], [RFC6438]) encourage IPv6 endpoints to set a non-zero value in this field 💜. A multiplexing transport could choose to use multiple flow labels, so the network can either forward its subflows independently or use one common value for the traffic aggregate. The flow label is present in all fragments. IPsec was originally put forward as one important use case for this mechanism, and it does encrypt the field [RFC6438] 🔐.

Once it's set, the flow label can tell network nodes about subflows at the transport layer, without them needing to interpret the settings of upper-layer protocol fields [RFC6294] ✨. That information can also help coordinate how aggregates of transport subflows get grouped when queued in the network, and select appropriate per-flow forwarding when choosing between alternate paths [RFC6438]. Examples are Equal-Cost Multipath (ECMP) routing and Link Aggregation Groups (LAGs) 🦋.

6.8.1. Reasons for Non-deployment ✨

okay so even though the flow label is sitting in every single IPv6 packet 🎀, it did not get anywhere near as much use as it was supposed to. 🥺 The main reason is that it only becomes useful when two different stakeholders both show up and do their part 🫶

  • Endpoint Implementation:

ok so for network nodes along a path to actually use the flow label, there needs to be a non-zero value put in the field [RFC6437] at the sending endpoint 💅 and there needs to be an incentive for the endpoint to set an appropriate non-zero value. 💖 The value should appropriately reflect the level of aggregation the traffic is expected to get from the network. 🌸 BUT this means the stack has to know the granularity at which flows should be identified (or, the other way round, which flows should get aggregated treatment), i.e., which packets carry the same flow label 😭 So setting a non-zero value may lead to extra choices that an application developer has to make. 🙈

Also, the original flow label standard [RFC3697] forbids encoding any meaning into the flow label value 🚫 but the chance to use the flow label as a covert channel or to signal other meta-information may have raised concerns about setting a non-zero value [RFC6437]. 👀 not me being like "wait, that's a lot of risk" 💀

Before methods are widely deployed to use this, there could be no incentive for an endpoint to set the field. 🥀

  • Operational support in network nodes:

a benefit only happens when a network node along the path also uses this information to inform its decisions 🧚 Network equipment (routers and/or middleboxes) needs appropriate support in order to use the field when deciding how to classify flows or forward packets 🦋 The use of any optional feature in a network node also needs corresponding updates to operational procedures, so it is normally only introduced when the cost can be justified 💸

A benefit from using the flow label is expected to be increased quality of experience for applications 🌈 but this comes with some operational cost to an operator, and it needs endpoints to set the field too. 😤

6.8.2. Lessons Learned 🍓

the flow label is a general-purpose header field for use by the path, and multiple uses have been proposed 🧠 One candidate use was to reduce the complexity of forwarding decisions. 🤔 However, modern routers can use a "fast path", often taking advantage of hardware to speed up processing ⚡ The method can help with more complex forwarding, such as ECMP routing and load balancing. 👑

Although [RFC6437] recommended that endpoints should by default choose uniformly distributed labels for their traffic, the specification let an endpoint choose to set a zero value 🙃 This ability for endpoints to choose a flow label of zero has had consequences on deployability, and it's giving "plot twist" 😭

  • Before wide-scale support by endpoints, it would be impossible to rely on a non-zero flow label being set 💔 So network nodes would also need to use other techniques to get the same functions. One example is a method that assumes the semantics of the source port field to provide entropy input to a network-layer hash. 🧩 Using a 5-tuple to classify a packet is a layering violation [RFC6294] 🙅‍♀️ When these other methods have been deployed, they increase the cost of deploying standards-based methods, even though they may offer less control to endpoints and cause potential interaction with other uses or interpretations of the field. 🥴
  • Even though the flow label is specified as an end-to-end field, some network paths have been observed to not transparently forward the flow label 😳 This could be because of non-conformant equipment, or it could mean some operational networks have chosen to reuse the protocol field for other (e.g., internal) purposes 🕵️‍♀️ This leads to a lack of transparency, and it's a deployment hurdle for endpoints that expect to set a flow label the network will actually use 🚧 The more recent practice of "greasing" [GREASE] suggests a different outcome could have been achieved if endpoints were always required to set a non-zero value. 💅✨
  • [RFC1809] noted that choosing the flow label value can depend on the expectations of the traffic generated by an application, which suggests that an API should be presented to control the setting or the policy that is used 🎛️ However, many currently available APIs don't have this support. 😭

A growth in the use of encrypted transports (e.g., QUIC [RFC9000]) seems likely to raise issues similar to those discussed above, and it could motivate renewed interest in using the flow label 🔐🦋 main character return arc incoming 👑

6.9. Explicit Congestion Notification (ECN) 🌈

The suggested references for Explicit Congestion Notification (ECN) are:

  • "Recommendations on Queue Management and Congestion Avoidance in the Internet" [RFC2309] 📚
  • "A Proposal to add Explicit Congestion Notification (ECN) to IP" [RFC2481] 📚
  • "The Addition of Explicit Congestion Notification (ECN) to IP" [RFC3168] 📚
  • "Implementation Report on Experiences with Various TCP RFCs" [vista-impl], slides 6 and 7 📚
  • "Implementation and Deployment of ECN" (at [SallyFloyd]) 📚

In the early 1990s, the large majority of Internet traffic used TCP as its transport protocol 🌸 but TCP had no way to detect path congestion before the path was so congested that packets were being dropped 😩 These congestion events could affect all senders using a path, either by "lockout", where long-lived flows monopolized the queues along a path, or by "full queues", where queues stay full, or almost full, for a long period of time. 🥶

In response, "Active Queue Management" (AQM) was deployed in the network 🛠️ A number of AQM disciplines have been deployed, but one common approach was that routers dropped packets when a threshold buffer length was reached, so transport protocols like TCP that respond to loss would detect the loss and reduce their sending rates 📉 Random Early Detection (RED) was one such proposal in the IETF. 🎀 As the name suggests, a router using RED as its AQM discipline that detected time-averaged queue lengths passing a threshold would probabilistically choose incoming packets to be dropped [RFC2309]. 🎲

Researchers suggested providing "explicit congestion notifications" to senders when routers along the path detected that their queues were building 🚦 This gives senders an opportunity to "slow down" as if a loss had occurred, giving path queues time to drain while the path still has enough buffer capacity to handle bursty arrivals of packets from other senders 🧘‍♀️ This was proposed as an experiment in [RFC2481] and standardized in [RFC3168]. 🎉

A key aspect of ECN was using IP header fields rather than IP options to carry explicit congestion notifications 🧠 since the proponents recognized that

Many routers process the "regular" headers in IP packets more efficiently than they process the header information in IP options.

Unlike most of the Path Aware technologies included in this document, the story of ECN continues to the present day and has encountered a large number of Lessons Learned along the way 🕰️✨ The early history of ECN (non-)deployment provides Lessons Learned that were not captured by other contributions in Section 6, so that is the emphasis of this section of the document. 💜

6.9.1. Reasons for Non-deployment 🥹

ECN deployment relied on three factors: support in client implementations, support in router implementations, and deployment decisions in operational networks 🧩

The proponents of ECN did so much right 👏 anticipating many of the Lessons Learned now recognized in Section 4. They recognized the need to support incremental deployment (Section 4.2). They considered the impact on router throughput (Section 4.8). They even considered trust issues between end nodes and the network, for both non-compliant end nodes (Section 4.10) and non-compliant routers (Section 4.9). ✨ literally the goat behavior 🐐

They were rewarded with ECN being implemented in major operating systems, for both end nodes and routers 🥳 A number of implementations are listed under "Implementation and Deployment of ECN" at [SallyFloyd]. 📋

What they did not anticipate was routers that would crash when they saw bits 6 and 7 in the IPv4 Type of Service (TOS) octet [RFC0791] / IPv6 Traffic Class field [RFC2460] being set to a non-zero value, which [RFC2481] had redefined as "Currently Unused" 💀😭 bestie, nobody saw this coming 🙈

As described in [vista-impl] ("IGD" stands for "Intermediate Gateway Device"):

|  IGD problem #1: one of the most popular versions from one of the
|  most popular vendors.  When a data packet arrives with either
|  ECT(0) or ECT(1) (indicating successful ECN capability
|  negotiation) indicated, router crashed.  Cannot be recovered at
|  TCP layer [sic]

This implementation, which would be run on a significant percentage of Internet end nodes, shipped with ECN disabled, as was true for several of the other implementations listed under "Implementation and Deployment of ECN" at [SallyFloyd]. 🙃 Even after subsequent router vendors fixed these implementations, ECN was still disabled on end nodes 🛑 and given the trade-off between the benefits of enabling ECN (somewhat better behavior during congestion) and the risks of enabling ECN (possibly crashing a router somewhere along the path), ECN tended to stay disabled on implementations that supported it for decades afterwards. 🥀 the way that's so relatable tbh 😩🫶

6.9.2. Lessons Learned ✨

Okay so ECN is literally the ultimate cautionary tale 😭 Out of everything in Section 6, it's the one that gives us these lessons:

  • Even if you do everything right, you may trip over implementation bugs in devices you know nothing about 💀, and those bugs will cause severe problems that prevent successful deployment of your Path Aware technology.
  • After implementations disable your Path Aware technology, it may take years, or even decades, to convince implementers to re-enable it by default 🥲

Put together, these two lessons are basically "you get one chance to get it right" 🫠

During the ECN discussion at [PANRG-110], we noticed that "you get one chance to get it right" isn't quite correct anymore 👀 Operating systems on so many hosts get updated constantly, and transport protocols like QUIC [RFC9000] are implemented in user space, so they can be updated without touching the installed operating system. Neither of these factors were true in the early 2000s.

Here's how we'd restate the ECN Lessons Learned for current implementers, and honestly these are way more useful 🎀

  • Even if you do everything right, you may trip over implementation bugs in devices you know nothing about 💀, and those bugs will cause severe problems that prevent successful deployment of your Path Aware technology. Testing before deployment isn't enough to ensure successful deployment. You also need to "deploy gently" 🫶 which often means deploying to a small subset of users so you can gain experience, and implementing feedback mechanisms to detect when user experience is getting degraded.
  • After implementations disable your Path Aware technology, it may take years, or even decades, to convince implementers to re-enable it by default. This might be based on the difficulty of distributing implementations that enable it by default, but it's just as likely to be based on the "bad taste in the mouth" 😖 that implementers have after an unsuccessful deployment attempt that degraded user experience.

With these expansions, the two lessons taken together can be summarized more helpfully as "plan for failure" 💅 In other words, anticipate what your next step will be if the initial deployment doesn't work out.

ECN deployment was also held back by the non-deployment of AQM in many devices 🙃 That happened because operators were interested in QoS features provided in the network, rather than using the network to help end systems provide for themselves. But that's a whole other story, and the AQM Lessons Learned are already covered in other contributions in Section 6 📚

7. Security Considerations 🔐

This document describes Path Aware techniques that were not adopted and widely deployed on the Internet, so it doesn't affect the security of the Internet 🦋

If this document meets its goals, we may develop new techniques for Path Aware networking that would affect the security of the Internet 👀 Those techniques will come with security considerations, which will be described in the corresponding RFCs that specify them. Stay tuned bestie 🌈

8. IANA Considerations

This document has no IANA actions 🧚

9. Informative References

   [RFC2215]  Shenker, S. and J. Wroclawski, "General Characterization
              Parameters for Integrated Service Network Elements",
              RFC 2215, DOI 10.17487/RFC2215, September 1997,
              <https://www.rfc-editor.org/info/rfc2215>.

   [RFC2309]  Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering,
              S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G.,
              Partridge, C., Peterson, L., Ramakrishnan, K., Shenker,
              S., Wroclawski, J., and L. Zhang, "Recommendations on
              Queue Management and Congestion Avoidance in the
              Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998,
              <https://www.rfc-editor.org/info/rfc2309>.

   [RFC2460]  Deering, S. and R. Hinden, "Internet Protocol, Version 6
              (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460,
              December 1998, <https://www.rfc-editor.org/info/rfc2460>.

   [RFC2475]  Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
              and W. Weiss, "An Architecture for Differentiated
              Services", RFC 2475, DOI 10.17487/RFC2475, December 1998,
              <https://www.rfc-editor.org/info/rfc2475>.

   [RFC2481]  Ramakrishnan, K. and S. Floyd, "A Proposal to add Explicit
              Congestion Notification (ECN) to IP", RFC 2481,
              DOI 10.17487/RFC2481, January 1999,
              <https://www.rfc-editor.org/info/rfc2481>.

   [RFC3168]  Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
              of Explicit Congestion Notification (ECN) to IP",
              RFC 3168, DOI 10.17487/RFC3168, September 2001,
              <https://www.rfc-editor.org/info/rfc3168>.

   [RFC3697]  Rajahalme, J., Conta, A., Carpenter, B., and S. Deering,
              "IPv6 Flow Label Specification", RFC 3697,
              DOI 10.17487/RFC3697, March 2004,
              <https://www.rfc-editor.org/info/rfc3697>.

   [RFC4094]  Manner, J. and X. Fu, "Analysis of Existing Quality-of-
              Service Signaling Protocols", RFC 4094,
              DOI 10.17487/RFC4094, May 2005,
              <https://www.rfc-editor.org/info/rfc4094>.

   [RFC4443]  Conta, A., Deering, S., and M. Gupta, Ed., "Internet
              Control Message Protocol (ICMPv6) for the Internet
              Protocol Version 6 (IPv6) Specification", STD 89,
              RFC 4443, DOI 10.17487/RFC4443, March 2006,
              <https://www.rfc-editor.org/info/rfc4443>.

   [RFC4782]  Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick-
              Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782,
              January 2007, <https://www.rfc-editor.org/info/rfc4782>.

   [RFC5082]  Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C.
              Pignataro, "The Generalized TTL Security Mechanism
              (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007,
              <https://www.rfc-editor.org/info/rfc5082>.

   [RFC5218]  Thaler, D. and B. Aboba, "What Makes for a Successful
              Protocol?", RFC 5218, DOI 10.17487/RFC5218, July 2008,
              <https://www.rfc-editor.org/info/rfc5218>.

   [RFC5533]  Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming
              Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533,
              June 2009, <https://www.rfc-editor.org/info/rfc5533>.

   [RFC5575]  Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J.,
              and D. McPherson, "Dissemination of Flow Specification
              Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009,
              <https://www.rfc-editor.org/info/rfc5575>.

   [RFC5681]  Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
              Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
              <https://www.rfc-editor.org/info/rfc5681>.

   [RFC5971]  Schulzrinne, H. and R. Hancock, "GIST: General Internet
              Signalling Transport", RFC 5971, DOI 10.17487/RFC5971,
              October 2010, <https://www.rfc-editor.org/info/rfc5971>.

   [RFC5973]  Stiemerling, M., Tschofenig, H., Aoun, C., and E. Davies,
              "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)",
              RFC 5973, DOI 10.17487/RFC5973, October 2010,
              <https://www.rfc-editor.org/info/rfc5973>.

   [RFC5974]  Manner, J., Karagiannis, G., and A. McDonald, "NSIS
              Signaling Layer Protocol (NSLP) for Quality-of-Service
              Signaling", RFC 5974, DOI 10.17487/RFC5974, October 2010,
              <https://www.rfc-editor.org/info/rfc5974>.

   [RFC5981]  Manner, J., Stiemerling, M., Tschofenig, H., and R. Bless,
              Ed., "Authorization for NSIS Signaling Layer Protocols",
              RFC 5981, DOI 10.17487/RFC5981, February 2011,
              <https://www.rfc-editor.org/info/rfc5981>.

   [RFC6294]  Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for
              the IPv6 Flow Label", RFC 6294, DOI 10.17487/RFC6294, June
              2011, <https://www.rfc-editor.org/info/rfc6294>.

   [RFC6398]  Le Faucheur, F., Ed., "IP Router Alert Considerations and
              Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October
              2011, <https://www.rfc-editor.org/info/rfc6398>.

   [RFC6437]  Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme,
              "IPv6 Flow Label Specification", RFC 6437,
              DOI 10.17487/RFC6437, November 2011,
              <https://www.rfc-editor.org/info/rfc6437>.

   [RFC6438]  Carpenter, B. and S. Amante, "Using the IPv6 Flow Label
              for Equal Cost Multipath Routing and Link Aggregation in
              Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011,
              <https://www.rfc-editor.org/info/rfc6438>.

   [RFC6633]  Gont, F., "Deprecation of ICMP Source Quench Messages",
              RFC 6633, DOI 10.17487/RFC6633, May 2012,
              <https://www.rfc-editor.org/info/rfc6633>.

   [RFC6928]  Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis,
              "Increasing TCP's Initial Window", RFC 6928,
              DOI 10.17487/RFC6928, April 2013,
              <https://www.rfc-editor.org/info/rfc6928>.

   [RFC7305]  Lear, E., Ed., "Report from the IAB Workshop on Internet
              Technology Adoption and Transition (ITAT)", RFC 7305,
              DOI 10.17487/RFC7305, July 2014,
              <https://www.rfc-editor.org/info/rfc7305>.

   [RFC7418]  Dawkins, S., Ed., "An IRTF Primer for IETF Participants",
              RFC 7418, DOI 10.17487/RFC7418, December 2014,
              <https://www.rfc-editor.org/info/rfc7418>.

   [RFC8085]  Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
              Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
              March 2017, <https://www.rfc-editor.org/info/rfc8085>.

   [RFC8170]  Thaler, D., Ed., "Planning for Protocol Adoption and
              Subsequent Transitions", RFC 8170, DOI 10.17487/RFC8170,
              May 2017, <https://www.rfc-editor.org/info/rfc8170>.

   [RFC8655]  Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", RFC 8655,
              DOI 10.17487/RFC8655, October 2019,
              <https://www.rfc-editor.org/info/rfc8655>.

   [RFC8793]  Wissingh, B., Wood, C., Afanasyev, A., Zhang, L., Oran,
              D., and C. Tschudin, "Information-Centric Networking
              (ICN): Content-Centric Networking (CCNx) and Named Data
              Networking (NDN) Terminology", RFC 8793,
              DOI 10.17487/RFC8793, June 2020,
              <https://www.rfc-editor.org/info/rfc8793>.

   [RFC9000]  Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
              Multiplexed and Secure Transport", RFC 9000,
              DOI 10.17487/RFC9000, May 2021,
              <https://www.rfc-editor.org/info/rfc9000>.

   [SAAG-105-Min]
              "Security Area Open Meeting - IETF 105 Minutes", July
              2019, <https://datatracker.ietf.org/meeting/105/materials/
              minutes-105-saag-00>.

   [SAF07]    Sarolahti, P., Allman, M., and S. Floyd, "Determining an
              appropriate sending rate over an underutilized network
              path", Computer Networks: The International Journal of
              Computer and Telecommunications Networking, Volume 51,
              Number 7, DOI 10.1016/j.comnet.2006.11.006, May 2007,
              <https://dl.acm.org/doi/10.1016/j.comnet.2006.11.006>.

   [SallyFloyd]
              Floyd, S., "ECN (Explicit Congestion Notification) in TCP/
              IP", June 2009, <https://www.icir.org/floyd/ecn.html>.

   [Sch11]    Scharf, M., "Fast Startup Internet Congestion Control for
              Broadband Interactive Applications", Ph.D. Thesis,
              University of Stuttgart, April 2011.

   [Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB
              IPv6 Multihoming Panel at NANOG 35", North American
              Network Operators' Group (NANOG), October 2005,
              <https://www.youtube.com/watch?v=ji6Y_rYHAQs>.

   [TRIGTRAN-55]
              "Triggers for Transport BOF at IETF 55", November 2002,
              <https://www.ietf.org/proceedings/55/239.htm>.

   [TRIGTRAN-56]
              "Triggers for Transport BOF at IETF 56", March 2003,
              <https://www.ietf.org/proceedings/56/251.htm>.

   [vista-impl]
              Sridharan, M., Bansal, D., and D. Thaler, "Implementation
              Report on Experiences with Various TCP RFCs", March 2007,
              <https://www.ietf.org/proceedings/68/slides/tsvarea-3/
              sld1.htm>.

Acknowledgments 🫶

Gorry Fairhurst brought the initial material for Section 6.1 on ST2 ✨

Ron Bonica brought the initial material for Section 6.2 on IntServ 🌸

Michael Scharf wrote the initial material for Section 6.3 on Quick-Start TCP, and then also gave us suggestions to make that section even better after it got edited 💅

Gorry Fairhurst is back with the initial material for Section 6.4 on ICMP Source Quench 🦋

Spencer Dawkins wrote the initial material for Section 6.5 on Triggers for Transport (TRIGTRAN) 🔥

Erik Nordmark provided the obstacles that Section 6.6 on Shim6 is built on, and Spencer Dawkins added the background 🧚

Roland Bless and Martin Stiemerling provided the initial material for Section 6.7 on Next Steps in Signaling (NSIS) 💜

Gorry Fairhurst provided the initial material for Section 6.8 on IPv6 Flow Labels 🎀

Spencer Dawkins provided the initial material for Section 6.9 on Explicit Congestion Notification 🌈

Huge thanks to Adrian Farrel, Bob Briscoe, C.M. Heard, David Black, Eric Kinnear, Erik Auerswald, Gorry Fairhurst, Jake Holland, Joe Touch, Joeri de Ruiter, Kireeti Kompella, Mohamed Boucadair, Randy Presuhn, Roland Bless, Ruediger Geib, Theresa Enghardt, and Wes Eddy, who reviewed this document as a "work in process" and left comments 🙈

Mallory Knodel reviewed this document for the Internet Research Steering Group and gave us lots of helpful suggestions 👑

David Oran also gave helpful comments and text suggestions on this document during Internet Research Steering Group balloting. In particular, Section 5 reflects his review 💖

Benjamin Kaduk, Martin Duke, and Rob Wilton gave helpful comments during Internet Engineering Steering Group conflict review 🍓

Special thanks to Adrian Farrel for helping Spencer navigate the twisty little passages of Flow Specs and Filter Specs in IntServ, RSVP, MPLS, and BGP. They are all alike, except when they are different [Colossal-Cave]. 💀

Author's Address 🎀

Spencer Dawkins (editor) Tencent America United States of America

Email: spencerdawkins.ietf@gmail.com 💌

📦 where this page comes from