RFC 3246

An Expedited Forwarding PHB (Per-Hop Behavior)

📅 published March 2002, from ietf

RFC 3246: An Expedited Forwarding PHB (Per-Hop Behavior) 💖🎀

Okay bestie, this is RFC 3246 (Part 1 of 3) ✨ It's a Standards Track doc from the Network Working Group, dropped in March 2002, and it defines the Expedited Forwarding PHB. The authors are a whole squad: B. Davie and A. Charny from Cisco Systems, Inc., plus J.C.R. Bennett (Motorola), K. Benson (Tellabs), J.Y. Le Boudec (EPFL), W. Courtney (TRW), S. Davari (PMC-Sierra), V. Firoiu (Nortel Networks), and D. Stiliadis (Lucent Technologies). It obsoletes RFC 2598 🦋 and the copyright belongs to The Internet Society (2001).

Status of this Memo 📜

This is an Internet standards track protocol for the Internet community, and it's asking for discussion and suggestions for improvements 💅 To check the standardization state and status of this protocol, look at the current edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited.

Copyright (C) The Internet Society (2001). All Rights Reserved.

Abstract 🧚

So this doc defines a PHB (per-hop behavior) called Expedited Forwarding (EF) 🔥 The PHB is a basic building block in the Differentiated Services architecture. EF is meant to be a building block for low delay, low jitter, and low loss services, and it does that by making sure the EF aggregate is served at a certain configured rate ⭐ This document obsoletes RFC 2598.

Table of Contents 🍓

1      Introduction  ...........................................   2
1.1    Relationship to RFC 2598  ...............................   3
2      Definition of EF PHB  ...................................   3
2.1    Intuitive Description of EF  ............................   3
2.2    Formal Definition of the EF PHB  ........................   5
2.3    Figures of merit  .......................................   8
2.4    Delay and jitter  .......................................   8
2.5    Loss  ...................................................   9
2.6    Microflow misordering  ..................................   9
2.7    Recommended codepoint for this PHB  .....................   9
2.8    Mutability  .............................................  10
2.9    Tunneling  ..............................................  10
2.10   Interaction with other PHBs  ............................  10
3      Security Considerations  ................................  10
4      IANA Considerations  ....................................  11
5      Acknowledgments  ........................................  11
6      References  .............................................  11
Appendix: Implementation Examples ..............................  12
Authors' Addresses .............................................  14
Full Copyright Statement .......................................  16

1. Introduction 🦋

Okay so network nodes that implement the differentiated services enhancements to IP use a codepoint in the IP header to pick a per-hop behavior (PHB), which is the specific forwarding treatment for that packet [3, 4]. This memo describes one particular PHB called expedited forwarding (EF) 💖

The whole point of the EF PHB is to be a building block for low loss, low delay, and low jitter services ✨ How you actually build those services is outside the scope of this spec, so that's a different convo 🙈

Here's the tea on delay: the dominant causes are fixed propagation delays (like the ones from speed-of-light travel) on wide area links, plus queuing delays in switches and routers 🔥 Propagation delay is a fixed property of the topology, so delay and jitter get minimized when queuing delays get minimized. In this context, jitter is defined as the variation between maximum and minimum delay. The vibe of the EF PHB is that suitably marked packets usually run into short or empty queues. And if queues stay short compared to the buffer space available, packet loss stays low too 💅

To keep EF queues usually short, the service rate of EF packets on a given output interface has to exceed their arrival rate at that interface over long and short time intervals, no matter how much other (non-EF) traffic is going on 🌈 This spec defines a PHB where EF packets are guaranteed to get service at or above a configured rate, and it gives you a way to quantify how accurately that service rate gets delivered over any time interval. It also gives you a way to quantify the maximum delay and jitter a packet may experience under bounded operating conditions 🫶

Note that the EF PHB only defines the behavior of a single node 👑 The behavior of a whole collection of nodes is outside the scope of this doc. A Per-Domain Behavior (PDB) specification [7] may provide that info.

When a DS-compliant node claims to implement the EF PHB, the implementation MUST conform to the specification given in this document. But here's the plot twist: the EF PHB is not a mandatory part of the Differentiated Services architecture. A node is NOT REQUIRED to implement the EF PHB in order to be considered DS-compliant 🙈

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [2]. 📜

1.1. Relationship to RFC 2598 🎀

This document replaces RFC 2598 [1]. The main difference is that it adds mathematical formalism to give a more rigorous definition of the behavior described 🧚 The full rationale for that is given in [6].

2. Definition of EF PHB 💜

2.1. Intuitive Description of EF ✨

Intuitively, EF is actually simple. The rate at which EF traffic gets served at a given output interface should be at least the configured rate R, over a suitably defined interval, no matter what the offered load of non-EF traffic to that interface is 🦋 Two difficulties show up when we try to formalize that vibe:

  • It's hard to define the right timescale to measure R on 😭 If you measure at short timescales, you may introduce sampling errors. If you measure at long timescales, you may allow excessive jitter.
  • EF traffic clearly can't be served at rate R if there are no EF packets waiting to be served. The problem is it may be impossible to tell from the outside whether EF packets are actually waiting for the output scheduler. For example, if an EF packet has entered the router and not exited yet, it may be awaiting service, or it may have just hit some processing or transmission delay inside the router 💀

The formal definition below takes these issues into account 💅 It assumes that EF packets should ideally be served at rate R or faster, and it bounds how far the actual departure time of each packet can deviate from that packet's "ideal" departure time. We define the departure time of a packet as the time when the last bit of that packet leaves the node 🌸 The "ideal" departure time of each EF packet gets computed iteratively.

When an EF packet arrives at a device and all the previous EF packets have already departed, the computation of the ideal departure time is simple 🍓 Service of the packet should (ideally) start as soon as it arrives, so the ideal departure time is just the arrival time plus the ideal time to transmit the packet at rate R. For a packet of length L_j, that transmission time at the configured rate R is L_j/R. (Of course, a real packet will typically get transmitted at line rate once its transmission actually starts, but we're calculating the ideal target behavior here, so the ideal service takes place at rate R. 🧚)

When an EF packet arrives at a device that still has EF packets awaiting service, it gets more complicated 🔥 There are two cases to think about. If the previous (j-1-th) departure happened after its own ideal departure time, then the scheduler is running "late". In that case, the ideal time to start service of the new packet is the ideal departure time of the previous (j-1-th) packet, or the arrival time of the new packet, whichever is later, because we can't expect a packet to begin service before it arrives 🙈 If the previous (j-1-th) departure happened before its own ideal departure time, then the scheduler is running "early". In that case, service of the new packet should begin at the actual departure time of the previous packet ✨

Once we know when service of the j-th packet should (ideally) begin, the ideal departure time of the j-th packet is L_j/R seconds later 👑 So we can express the ideal departure time of the j-th packet in terms of the arrival time of the j-th packet, the actual departure time of the j-1-th packet, and the ideal departure time of the j-1-th packet. Equations eq_1 and eq_2 in Section 2.2 capture this relationship 🦋

Okay, here's where it gets extra: the original EF definition didn't give any way to guarantee the delay of an individual EF packet, but that property might be desired 💖 So the equations in Section 2.2 have two parts: an "aggregate behavior" set and a "packet-identity-aware" set. The aggregate behavior equations (eq_1 and eq_2) just describe the properties of the service the device delivers to the EF aggregate. The "packet-identity-aware" equations (eq_3 and eq_4) let you calculate the bound on delay of an individual packet, given knowledge of the device's operating conditions 🌈 The significance of these two sets is discussed further in Section 2.2. Note that these two sets of equations give two ways of characterizing the behavior of a single device, not two different modes of behavior. No cap, that's the key takeaway 🫶

2.2. Formal Definition of the EF PHB 💖

okay bestie, this is the math part and i'm obsessed 🧚 A node that supports EF on an interface I at some configured rate R MUST satisfy these equations:

   d_j <= f_j + E_a for all j > 0                             (eq_1)

where f_j is built up step by step like this:

   f_0 = 0, d_0 = 0

   f_j = max(a_j, min(d_j-1, f_j-1)) + l_j/R,  for all j > 0  (eq_2)

Here's what all the bits mean 🎀

  • d_j is the time the last bit of the j-th EF packet to depart actually leaves the node from interface I.
  • f_j is the target departure time for the j-th EF packet to depart from I, the "ideal" time by which the last bit of that packet should be out of the node.
  • a_j is the time the last bit of the j-th EF packet destined for output I actually shows up at the node.
  • l_j is the size (in bits) of the j-th EF packet to depart from I. It's measured on the IP datagram (IP header plus payload) and does NOT include any lower layer (e.g. MAC layer) overhead 🙅‍♀️
  • R is the EF configured rate at output I (in bits/second).
  • E_a is the error term for how the EF aggregate gets treated. It's the worst case gap between when an EF packet actually leaves and when it ideally should have, so E_a gives an upper bound on (d_j - f_j) for all j.
  • d_0 and f_0 aren't real packet departures, they're just there so the recursion works. The time origin should be picked so that no EF packets are in the system at time 0.
  • for a_j and d_j, the "last bit" of the packet includes the layer 2 trailer if there is one, because a packet generally can't be considered available for forwarding until that trailer has been received.

An EF-compliant node MUST be able to be characterized by the range of possible R values it can support on each of its interfaces while following these equations, plus the value of E_a it can hit on each interface. R may be line rate or less. E_a MAY be specified as a worst-case value for all possible R values, or MAY be expressed as a function of R ✨

Also, a node can have multiple inputs and some pretty complex internal scheduling, so the j-th EF packet to arrive at the node for a certain interface might NOT be the j-th EF packet to leave that interface. That's the sense in which eq_1 and eq_2 are unaware of packet identity 🫶

In addition, a node that supports EF on an interface I at some configured rate R MUST satisfy these equations:

   D_j <= F_j + E_p for all j > 0                             (eq_3)

where F_j is built up like this:

   F_0 = 0, D_0 = 0

   F_j = max(A_j, min(D_j-1, F_j-1)) + L_j/R,  for all j > 0  (eq_4)

Here's the breakdown 💅

  • D_j is the actual departure time of the individual EF packet that arrived at the node for interface I at time A_j. So, given the packet that was the j-th EF packet for I to arrive at the node through any input, D_j is the time the last bit of that individual packet actually leaves the node from I.
  • F_j is the target departure time for the individual EF packet that arrived at the node for interface I at time A_j.
  • A_j is the time the last bit of the j-th EF packet destined for output I to arrive actually arrives at the node.
  • L_j is the size (in bits) of the j-th EF packet to arrive at the node that is destined for output I. It's measured on the IP datagram (IP header plus payload) and does NOT include any lower layer (e.g. MAC layer) overhead.
  • R is the EF configured rate at output I (in bits/second).
  • E_p is the error term for how individual EF packets get treated. It's the worst case gap between when an EF packet actually leaves and when it ideally should have, so E_p gives an upper bound on (D_j - F_j) for all j.
  • D_0 and F_0 aren't real packet departures either, they're just for the recursion. The time origin should be picked so that no EF packets are in the system at time 0.
  • for A_j and D_j, the "last bit" of the packet includes the layer 2 trailer if there is one, because a packet generally can't be considered available for forwarding until that trailer has been received.

It's the fact that D_j and F_j refer to the departure times for the j-th packet to arrive that makes eq_3 and eq_4 packet-aware. This is the big difference between the last two equations and the first two 🌈

An EF-compliant node SHOULD be able to be characterized by the range of possible R values it can support on each of its interfaces while following these equations, plus the value of E_p it can hit on each interface. E_p MAY be specified as a worst-case value for all possible R values, or MAY be expressed as a function of R. An E_p value of "undefined" MAY be specified. For the discussion of situations where E_p might be undefined, check the Appendix and [6] 📚

For testing conformance to these equations, you might have to deal with packet arrivals on different interfaces that are super close together in time. If two or more EF packets headed for the same output interface arrive (on different inputs) at almost the same moment, and the gap between their arrival times can't be measured, then it's fine to use a random tie-breaking method to decide which packet arrived "first" 🎲

2.3. Figures of merit 🦋

E_a and E_p can be thought of as "figures of merit" for a device. A smaller E_a means the device serves the EF aggregate more smoothly at rate R over relatively short timescales. A larger E_a implies a more bursty scheduler, one that serves the EF aggregate at rate R only when you measure over longer intervals. A device with a larger E_a can "fall behind" the ideal service rate R by a bigger amount than a device with a smaller E_a 😭

A lower E_p implies a tighter bound on the delay an individual packet experiences. Stuff that might push E_p higher includes a large number of input interfaces (since an EF packet might arrive just behind a bunch of EF packets that came in on other interfaces), or internal scheduler details (e.g. per-flow scheduling within the EF aggregate) 🌸

We observe that the factors that increase E_a (like the ones above) will also increase E_p, so E_p is typically greater than or equal to E_a. Short version: E_a measures how far the EF aggregate drifts from ideal service at rate R, while E_p measures both non-ideal service AND non-FIFO treatment of packets inside the aggregate 💜

For more on all this, check the Appendix and [6] 🫧

2.4. Delay and jitter ⭐

Once you know E_p and the bounds on the EF traffic offered to a given output interface (summed over all input interfaces), you can bound the delay and jitter that EF traffic experiences leaving the node through that interface. The delay bound is

   D = B/R + E_p          (eq_5)

where

  • R is the configured EF service rate on the output interface
  • the total offered load of EF traffic destined for the output interface, summed over all input interfaces, is bounded by a token bucket of rate r <= R and depth B

Since the minimum delay through the device is clearly at least zero, D also gives a bound on jitter. To get a tighter jitter bound, the value of E_p for a device MAY be specified as two separate components, like this:

   E_p = E_fixed + E_variable

where E_fixed is the minimum delay an EF packet can experience going through the node 🧚

2.5. Loss 🥺

The EF PHB is meant to be a building block for low loss services. BUT under a high enough load of EF traffic (including unexpectedly big bursts from lots of inputs at once), any device with finite buffers may need to throw packets away. So it has to be possible to establish whether a device conforms to the EF definition even when some packets are lost. This gets done with an "off-line" test of conformance to equations 1 through 4. After watching a sequence of packets enter and leave the node, the packets that didn't leave are assumed lost and are notionally removed from the input stream. The packets that remain are the arrival stream (the a_j's), and the packets that left the node are the departure stream (the d_j's). So conformance to the equations can be checked by only looking at the packets that made it through the node 🫶

To help hit the low loss goal of EF, a node MAY be characterized by the operating region in which loss of EF due to congestion will NOT occur. This MAY be specified, using a token bucket of rate r <= R and burst size B, as the sum of traffic across all inputs to a given output interface that can be tolerated without loss 🍓

If loss DOES happen, which packets get lost is beyond the scope of this document. HOWEVER, it is a requirement that the packets not lost MUST conform to the equations of Section 2.2 💀

2.6. Microflow misordering 🎀

Packets belonging to a single microflow within the EF aggregate passing through a device SHOULD NOT experience re-ordering during normal operation of the device 🌈

Codepoint 101110 is RECOMMENDED for the EF PHB. iykyk 💅

2.8. Mutability 🦋

Packets marked for the EF PHB MAY be remarked at a DS domain boundary, but ONLY to other codepoints that satisfy the EF PHB. Packets marked for EF PHBs SHOULD NOT be demoted or promoted to another PHB by a DS domain 🙈

2.9. Tunneling 🧚

When EF packets are tunneled, the tunneling packets SHOULD be marked as EF. A full discussion of tunneling issues is in [5] 🫶

2.10. Interaction with other PHBs 💜

Other PHBs and PHB groups may be deployed in the same DS node or domain as the EF PHB. The equations of Section 2.2 MUST hold for a node no matter how much non-EF traffic is offered to it 🔥

If the EF PHB is implemented by a mechanism that allows unlimited preemption of other traffic (e.g., a priority queue), the implementation MUST include some means to limit the damage EF traffic could inflict on other traffic (e.g., a token bucket rate limiter). Traffic that exceeds this limit MUST be discarded. This maximum EF rate, and burst size if appropriate, MUST be settable by a network administrator (using whatever mechanism the node supports for non-volatile configuration). This part is NOT optional, bestie 😭✨

3. Security Considerations 🔐✨

Okay bestie, security time, and this part is seriously important 😤. To protect itself from denial of service attacks (aka someone flooding your network just to be annoying 💀), the edge of a DS domain SHOULD strictly police all EF marked packets to a rate negotiated with the adjacent upstream domain. Packets over the negotiated rate SHOULD be dropped. If two adjacent domains never negotiated an EF rate, the downstream domain SHOULD use 0 as the rate, which means it drops every single EF marked packet 🙅‍♀️.

Also, traffic conditioning at the ingress to a DS-domain MUST ensure that only packets with DSCPs that correspond to an EF PHB when they enter the DS-domain get marked with a DSCP that corresponds to EF inside the DS-domain. This is what the Differentiated Services architecture [4] requires, and it's giving protection 🛡️. It guards against denial-of-service and theft-of-service attacks that abuse DSCPs which aren't identified in any Traffic Conditioning Specification provisioned at an ingress interface, but which map to EF inside the DS-domain.

Don't let random DSCPs sneak into EF, periodt 🚨

4. IANA Considerations 📋

Okay so this one's short and sweet 🍓. This document allocates one codepoint, 101110, in Pool 1 of the code space defined by [3]. That's the only thing being claimed here, no cap 💅.

5. Acknowledgments 🫶

This doc was basically a group project with a huge crew 😭. Fred Baker, Angela Chiu, Chuck Kalmanek, and K. K. Ramakrishnan made significant contributions to the new EF definition, which is iconic 👑. John Wroclawski gave the authors lots of helpful comments 💜. This document leans really heavily on the original EF PHB definition by Jacobson, Nichols, and Poduri. It was also greatly influenced by the work of the EFRESOLVE team: Armitage, Casati, Crowcroft, Halpern, Kumar, and Schnizlein 🌈.

6. References 📚

  • [1] Jacobson, V., Nichols, K. and K. Poduri, "An Expedited Forwarding PHB", RFC 2598, June 1999.
  • [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [3] Nichols, K., Blake, S., Baker, F. and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998.
  • [4] Black, D., Blake, S., Carlson, M., Davies, E., Wang, Z. and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998.
  • [5] Black, D., "Differentiated Services and Tunnels", RFC 2983, October 2000.
  • [6] Charny, A., Baker, F., Davie, B., Bennett, J.C.R., Benson, K., Le Boudec, J.Y., Chiu, A., Courtney, W., Davari, S., Firoiu, V., Kalmanek, C., Ramakrishnan, K.K. and D. Stiliadis, "Supplemental Information for the New Definition of the EF PHB (Expedited Forwarding Per-Hop Behavior)", RFC 3247, March 2002.
  • [7] Nichols K. and B. Carpenter, "Definition of Differentiated Services Per Domain Behaviors and Rules for their Specification", RFC 3086, April 2001.

Appendix: Implementation Examples 🧚

This appendix isn't part of the normative spec of EF, so think of it as bonus tips and not the rulebook 📖. It's included as a possible source of useful info for implementors 💖.

A bunch of factors inside a node that supports EF will shape the values of E_a and E_p 🤓. Those factors get a deeper dive in [6], and they include both output schedulers and the internal design of a device.

A priority queue is widely considered the canonical example of an EF implementation, and it's a total icon 👑. A "perfect" output buffered device (one that delivers packets straight to the right output queue, instantly) with a priority queue for EF traffic gives you both a low E_a and a low E_p ✨. The main thing that affects E_a is the inability to pre-empt an MTU-sized non-EF packet that has just started transmitting when an EF packet shows up at the output interface, plus any extra delay from non-pre-emptable queues between the priority queue and the physical interface. E_p is mostly influenced by the number of interfaces 🔌.

Another example is a weighted round robin scheduler 🎀. This kind of scheduler usually can't support values of R as high as the link speeds, since the maximum rate EF traffic can be served at, with competing traffic around, depends on how many other queues there are and what weights they get. It's also likely to have a higher E_a than a priority queue implementation, all else being equal, because the round robin scheduler spends time serving non-EF queues 🙃.

Finally, it's possible to build hierarchical scheduling algorithms 🦋. Here, some non-FIFO scheduling algorithm runs on sub-flows inside the EF aggregate, while the EF aggregate as a whole could be served at high priority or with a large weight by the top-level scheduler. Such an algorithm might do per-input scheduling or per-microflow scheduling within the EF aggregate, for example. Since these algorithms lead to non-FIFO service within the EF aggregate, the value of E_p for them may be higher than for other implementations. For some schedulers of this type it may be difficult to give a meaningful bound on E_p that holds for any pattern of traffic arrival, so a value of "undefined" may be most appropriate 🤷‍♀️.

It's quite acceptable for a Diffserv domain to provide multiple instances of EF 🌸. Each instance should be characterizable by the equations in Section 2.2 of this specification. How multiple instances affect the E_a and E_p values of each one depends a lot on how they're implemented 🧩. For example, in a multi-level priority scheduler, an instance of EF that isn't at the highest priority may go through relatively long periods with no service while higher priority instances of EF get served, which results in relatively large values of E_a and E_p. By contrast, in a WFQ-like scheduler, each instance of EF is represented by a queue served at some configured rate, and the values of E_a and E_p could be similar to those of a single EF instance 💜.

Authors' Addresses 🗺️

Bruce Davie
Cisco Systems, Inc.
300 Apollo Drive
Chelmsford, MA, 01824

EMail: bsd@cisco.com

Anna Charny
Cisco Systems
300 Apollo Drive
Chelmsford, MA 01824

EMail: acharny@cisco.com

Jon Bennett
Motorola
3 Highwood Drive East
Tewksbury, MA 01876

EMail: jcrb@motorola.com

Kent Benson
Tellabs Research Center
3740 Edison Lake Parkway #101
Mishawaka, IN  46545

EMail: Kent.Benson@tellabs.com

Jean-Yves Le Boudec
ICA-EPFL, INN
Ecublens, CH-1015
Lausanne-EPFL, Switzerland

EMail: jean-yves.leboudec@epfl.ch

Bill Courtney
TRW
Bldg. 201/3702
One Space Park
Redondo Beach, CA 90278

EMail: bill.courtney@trw.com

Shahram Davari
PMC-Sierra Inc
411 Legget Drive
Ottawa, ON K2K 3C9, Canada

EMail: shahram_davari@pmc-sierra.com

Victor Firoiu
Nortel Networks
600 Tech Park
Billerica, MA 01821

EMail: vfiroiu@nortelnetworks.com

Dimitrios Stiliadis
Lucent Technologies
101 Crawfords Corner Road
Holmdel, NJ 07733

EMail: stiliadi@bell-labs.com

The legal stuff, kept exactly as written 🙏:

Copyright (C) The Internet Society (2001).  All Rights Reserved.

This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works.  However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.

The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 💐

Funding for the RFC Editor function is currently provided by the Internet Society 💖.

📦 where this page comes from