RFC 5166
Metrics for the Evaluation of Congestion Control Mechanisms
RFC 5166: Metrics for the Evaluation of Congestion Control Mechanisms 📦✨
Network Working Group 🌐 Editor: S. Floyd, Ed. (Sally Floyd is the editor 💖) Category: Informational 🎀 (this is information, not an Internet standard, so no pressure 💅) Date: March 2008 📅 Obsoletes / Updates: this header doesn't list anything it obsoletes or updates, so it's a fresh start 🦋
This is part 1 of 6, so the story continues in the next part 🧚
Status of This Memo ✨
This memo is just here to share info with the Internet community. It's not an Internet standard, so nothing in here is a rule you have to follow 🫶 Distribution is unlimited, so share away 💜
IESG Note 🔥
Okay, important tea: this doc is not an IETF Internet Standard 🙈 It's the personal opinion of one or more members of the TMRG Research Group of the Internet Research Task Force. It might get considered for standardization by the IETF someday, or become an IRTF Research Group consensus document later 🌈
Abstract 🌸
So this doc is all about the metrics you should look at when you're checking out new or changed congestion control mechanisms for the Internet 📊 That covers brand-new transport protocols, tweaks to TCP, congestion control living at the application level, and Active Queue Management (AQM) mechanisms sitting in routers 🛣️ It's also the first in a whole series meant to level up the models we use to evaluate transport protocols 🧠✨
It comes from the Transport Modeling Research Group (TMRG) and got lots of detailed feedback from folks in the Research Group (RG) 💬 Here's the honest part: the research community (and the IETF, vendors, operators, and everyone else) does not all agree on what congestion control should be optimizing for. Think trade-offs between throughput and delay, fairness between competing flows, and so on 🤷♀️ But we do agree on one big thing: congestion control should be judged on a trade-off across a range of metrics, not by maxing out a single one 💅👑
Table of Contents 📚
1. Introduction ....................................................2
2. Metrics .........................................................3
2.1. Throughput, Delay, and Loss Rates ..........................4
2.1.1. Throughput ..........................................5
2.1.2. Delay ...............................................6
2.1.3. Packet Loss Rates ...................................6
2.2. Response Times and Minimizing Oscillations .................7
2.2.1. Response to Changes .................................7
2.2.2. Minimizing Oscillations .............................8
2.3. Fairness and Convergence ...................................9
2.3.1. Metrics for Fairness between Flows .................10
2.3.2. Metrics for Fairness between Flows with
Different Resource Requirements ....................10
2.3.3. Comments on Fairness ...............................12
2.4. Robustness for Challenging Environments ...................13
2.5. Robustness to Failures and to Misbehaving Users ...........14
2.6. Deployability .............................................14
2.7. Metrics for Specific Types of Transport ...................15
2.8. User-Based Metrics ........................................15
3. Metrics in the IP Performance Metrics (IPPM) Working Group .....15
4. Comments on Methodology ........................................16
5. Security Considerations ........................................16
6. Acknowledgements ...............................................16
7. Informative References .........................................171. Introduction ✨
Okay so, the whole point here is to get better at evaluating congestion control mechanisms, and this doc kicks things off by walking through some of the metrics you'd want to think about 💖 We also look at how the metrics relate to each other, like the super well-known trade-off between throughput and delay 🎢 Heads up, though: this doc does not try to list every metric a study could possibly care about. Researchers might have domain-specific metrics that go way beyond what's here 🧚
We look at metrics for aggregate traffic (so how flows affect the other traffic on the network) and also at the very different goals of different apps or transport protocols. Like, bulk data transfer wants high throughput, while interactive voice or video wants low delay 📞🎬 Different transport protocols or AQM mechanisms might care about different sets of metrics. One might be all about per-flow throughput, while another is built to hold up over wireless links, and they might care about fairness to competing traffic to different degrees ⚖️
We're hoping this doc gets used to evaluate proposed congestion control mechanisms across a wide range of metrics. Something like, "Mechanism X is great at Metric A, but it pays for that with bad performance on Metric B." 💅 The goal is a big-picture view of what's strong and what's weak about new proposals 🦋
There are follow-up documents planned that will lay out sets of simulation and testbed scenarios for evaluating transport protocols and congestion control mechanisms, based on what the research community currently considers best practice 🧪 These are not meant to be complete or final benchmark test suites. They're just one step among many for researchers to use 🌸 More follow-ups are also planned on how to actually use those scenarios in methodologies 📋
This document is a product of the Transport Modeling Research Group (TMRG), and it got detailed feedback from many members of the Research Group (RG) 💬 And yes, the same caveat applies here: there isn't necessarily a community-wide consensus (IETF, vendors, operations, or anyone else) on what congestion control should be designed to optimize, whether that's trade-offs between throughput and delay, fairness between competing flows, or whatever. What we do agree on is that congestion control should be evaluated on trade-offs across a range of metrics, not by optimizing a single one 👑
2. Metrics 🎀
These are the metrics we're going to talk about:
- Throughput;
- Delay;
- Packet loss rates;
- Response to sudden changes or to transient events;
- Minimizing oscillations in throughput or in delay;
- Fairness and convergence times;
- Robustness for challenging environments;
- Robustness to failures and to misbehaving users;
- Deployability;
- Metrics for specific types of transport;
- User-based metrics.
We'll go through each one below 💅 A lot of these metrics can be read in a few different ways: network-based, flow-based, or user-based. Network-based metrics might look at aggregate bandwidth and aggregate drop rates. Flow-based metrics might look at end-to-end transfer times for file transfers, or end-to-end delay and packet drop rates for interactive traffic. User-based metrics might look at how long a user waits or how satisfied they are with their multimedia experience 🙈 Our main goal here is just to explain which metrics can matter. We're not trying to decide the best methodology for using each general metric 🫶
For some metrics, like fairness, the network community doesn't really agree on what the goals should be. When that happens, the doc just tries to lay out the range of approaches people take 🌈
2.1. Throughput, Delay, and Loss Rates 💖
Throughput, delay, and loss rates have clear trade-offs with each other, so it helps to think about all three together 🎢 Those trade-offs are easiest to see in terms of queue management at the router. The question is: is the queue set up to maximize aggregate throughput, to minimize delay and loss rates, or to find some compromise between the two? 🤔 Another option is to look at a separate metric called power, defined here as throughput over delay, which combines throughput and delay into one number ⚡ That said, this document does not propose a clear target for the trade-off between throughput and delay. What we're really suggesting is that evaluations of transport protocols should explore the competing metrics 🦋
Instead of router-based metrics, you can use flow-based metrics, where the relationship between per-flow throughput, delay, and loss rates can often be given by the transport protocol itself ✨ For example, in TCP, the sending rate s in packets per second is given as:
s = 1.2/(RTT*sqrt(p)),where RTT is the round-trip time and p is the loss rate [FF99] 🧮
2.1.1. Throughput 🌊
Throughput can be measured as a router-based metric of aggregate link utilization, as a flow-based metric of per-connection transfer times, or as user-based metrics like utility functions or user wait times 📊 It's a clear goal for most congestion control mechanisms to maximize throughput, as long as they still respect application demand and the constraints from the other metrics 💅
Throughput is sometimes split up into throughput and goodput 🧁 Throughput is the link utilization or flow rate in bytes per second. Goodput is also measured in bytes per second, but it only counts the useful traffic, which is the subset of throughput that's actually useful 🎀 So "goodput" leaves out duplicate packets, packets that will get dropped downstream, packet fragments or ATM cells that the receiver drops because they can't be reassembled into complete packets, and similar stuff 🙈 In general, this document doesn't distinguish between throughput and goodput, and just uses the general term "throughput" 💜
Maximizing throughput matters in a wide range of environments, from super congested networks to underused ones, and from long-lived flows to really short ones 🌈 For example, throughput has been used as one of the metrics for evaluating Quick-Start, a proposal that lets flows start up faster than slow-start. The evaluation looked at throughput in terms of transfer times for connections across a range of transfer sizes [RFC4782] [SAF06] 🚀
Depending on the context, it might be enough to look at aggregate throughput or mean per-flow throughput [DM06]. In other contexts, you might need to look at the distribution of per-flow throughput instead 📉 Some researchers evaluate transport protocols by maximizing aggregate user utility, where a user's utility is generally defined as a function of that user's throughput [KMT98] 👑
Individual applications can have their own throughput needs too 💅 For example, real-time video can have really variable bandwidth demands, and Voice over IP (VoIP) traffic is sensitive to how much bandwidth it gets right after idle periods 📞 So user metrics for throughput can get more complicated than just the per-connection transfer time 🫶
🦋 Okay so that's the first chunk! Part 2 picks up with delay and the rest of the Section 2.1 metrics, so stay tuned, bestie 💖
2.1.2. Delay 🦋
okay so delay works like throughput: you can measure it as a router-based metric of queueing delay over time, or as a flow-based metric in terms of per-packet transfer times 💅. per-packet delay can also include the delay at the sender while it waits for the transport protocol to send the packet. and for reliable transfer, the per-packet transfer time the application sees includes any delay from retransmitting a lost packet 😭
bulk data transfer users might only care about per-packet transfer times insofar as they affect the per-connection transfer time 🌸. on the other end of the spectrum, for streaming media users, per-packet delay can be a huge concern 💀. also, the average delay might not capture what users actually care about! some users might care about the worst-case delay, or about the tail of the delay distribution 🙈
the iconic part: queueing delay at a router is shared by all flows at a FIFO (First-In First-Out) queue 🤯. so the router-based queueing delay caused by bulk data transfers can matter even if those bulk transfers don't care about per-packet transfer times at all ✨
2.1.3. Packet Loss Rates 🍓
packet loss rates can be measured as a network-based or a flow-based metric 🧚
when you're checking how packet losses or ECN marks (Explicit Congestion Notification) [RFC3168] affect the performance of a congestion control mechanism for an individual flow, researchers often use both the packet loss/mark rate for that connection and the congestion event rate (also called the loss event rate) 💜. a congestion event or loss event is one or more lost or marked packets in one round-trip time [RFC3448].
some users might care about packet loss rate only insofar as it affects per-connection transfer times, while others care about packet loss rates directly 👑. RFC 3611, RTP Control Protocol Extended Reports, describes a VoIP performance-reporting standard called RTP Control Protocol Extended Reports (RTCP XR), which includes a set of burst metrics. in RFC 3611, a burst is the maximal sequence starting and ending with a lost packet, and not including a sequence of Gmin or more packets that are not lost [RFC3611]. the burst metrics are: burst density (the fraction of packets in bursts), gap density (the fraction of packets in the gaps between bursts), burst duration (the mean duration of bursts in seconds), and gap duration (the mean duration of gaps in seconds) 🦋. RFC 3357 derives metrics for "loss distance" and "loss period", along with statistics that capture loss patterns experienced by packet streams on the Internet [RFC3357] ✨
sometimes it's useful to tell apart packets dropped at routers due to congestion from packets lost in the network due to corruption 🔥
one network-related reason to avoid high steady-state packet loss rates is to avoid congestion collapse in environments with paths containing multiple congested links. in those environments, high packet loss rates could mean congested links waste scarce bandwidth carrying packets that will only be dropped downstream before reaching the receiver [RFC2914] 😭. and ngl, the retransmit rate can be high, and the goodput correspondingly low, even with a low packet drop rate [AEO03] 💀
2.2. Response Times and Minimizing Oscillations 🎀
in this section we look at response times and oscillations together, because there are well-known trade-offs between improving response times and minimizing oscillations 💖. the scenarios that show the dangers of poor response times are also often quite different from the scenarios that show the dangers of unnecessary oscillations 🙈
2.2.1. Response to Changes 💅
one of the key concerns in congestion control design has been response times to sudden congestion in the network 🌈. on one hand, congestion control mechanisms should respond reasonably promptly to sudden congestion from routing or bandwidth changes, or from a burst of competing traffic. at the same time, congestion control mechanisms should not respond too severely to transient changes, e.g., to a sudden increase in delay that will dissipate in less than the connection's round-trip time 🫶
congestion control also has to contend with sudden changes in the bandwidth-delay product due to mobility 👑. such bandwidth-delay product changes are expected to become more frequent and to have greater impact than path changes today. because of both mobility and the heterogeneity of wireless access types (802.11b,a,g, WIMAX, WCDMA, HS-WCDMA, E-GPRS, Bluetooth, etc.), both the bandwidth and the round-trip delay can change suddenly, sometimes by several orders of magnitude 🤯
evaluating the response to sudden or transient changes is especially important for slowly responding congestion control mechanisms, such as equation-based congestion control [RFC3448] and AIMD (Additive Increase Multiplicative Decrease), or related mechanisms using parameters that make them more slowly-responding than TCP [BB01] [BBFS01] ✨
in addition to the responsiveness and smoothness of aggregate traffic, you can consider the trade-offs between responsiveness, smoothness, and aggressiveness for an individual connection [FHP00] [YKL01] 🌸. here, smoothness is defined by the largest reduction in the sending rate in one round-trip time, in a deterministic environment with a packet drop exactly every 1/p packets. responsiveness is defined as the number of round-trip times of sustained congestion required for the sender to halve the sending rate. aggressiveness is defined as the maximum increase in the sending rate in one round-trip time, in packets per second, in the absence of congestion. this aggressiveness can be a function of the mode of the transport protocol; for TCP, the aggressiveness of slow-start is quite different from the aggressiveness of congestion avoidance mode 💜
2.2.2. Minimizing Oscillations 🌊
one goal is stability, in terms of minimizing oscillations of queueing delay or of throughput 🧘. in practice, stability is frequently associated with rate fluctuations or variance. rate variations can result in fluctuations in router queue size, and therefore in queue overflows. these queue overflows can cause loss synchronizations across coexisting flows and periodic under-utilization of link capacity, both of which are considered to be general signs of network instability 😭. so measuring the rate variations of flows is often used to measure the stability of transport protocols 💜. to measure rate variations, [JWL04], [RX05], and [FHPW00] use the coefficient of variation (CoV) of per-flow transmission rates, and [WCL05] suggests using standard deviations of per-flow rates. since rate variations depend on time scales, it makes sense to measure them over various time scales ⏰
measuring per-flow rate variations, however, is only one aspect of transport protocol stability 🦋. a realistic experimental setting always involves multiple flows of the transport protocol being observed, along with a significant amount of cross traffic, with rates varying over time on both the forward and reverse paths. a congestion control protocol has to adapt its rate to the varying rates of competing traffic, so just measuring the per-flow statistics of a subset of the traffic could be misleading. that's because it measures rate fluctuations due in part to adaptation to competing traffic on the path 🙈. so per-flow statistics are most meaningful when they are accompanied by statistics measured at the network level. as a complementary metric to the per-flow statistics, [HKLRX06] uses measurements of the rate variations of the aggregate flows observed in bottleneck routers over various time scales 🔥
minimizing oscillations in queueing delay or throughput has related per-flow metrics of minimizing jitter in round-trip times and loss rates 🎀
an orthogonal goal for some congestion control mechanisms, e.g., for equation-based congestion control, is to minimize the oscillations in the sending rate for an individual connection, given an environment with a fixed, steady-state packet drop rate. (as is well known, TCP congestion control is characterized by a pronounced oscillation in the sending rate, with the sender halving the sending rate in response to congestion.) one metric for the level of oscillations is the smoothness metric given in Section 2.2.1 above 🌈
as discussed in [FK07], the synchronization of loss events can also affect convergence times, throughput, and delay 💖
2.3. Fairness and Convergence 👑
another set of metrics is fairness and convergence times ✨. fairness can be considered between flows of the same protocol and between flows using different protocols (e.g., TCP-friendliness, referring to fairness between TCP and a new transport protocol). fairness can also be considered between sessions, between users, or between other entities 🫶
there are a number of different fairness measures 🧁. these include max-min fairness [HG86], proportional fairness [KMT98] [K01], the fairness index proposed in [JCH84], and the product measure, a variant of network power [BJ81].
2.3.1. Metrics for Fairness between Flows 💅
this section discusses fairness metrics that consider the fairness between flows, but that don't take into account different characteristics of flows (e.g., the number of links in the path or the round-trip time). for the discussion of fairness metrics, let x_i be the throughput for the i-th connection 🌸
Jain's fairness index: the fairness index in [JCH84] is:
(( sum_i x_i )^2) / (n * sum_i ( (x_i)^2 )),where there are n users. this fairness index ranges from 0 to 1, and it is maximum when all users receive the same allocation. this index is k/n when k users equally share the resource, and the other n-k users receive zero allocation 🍓
The product measure: the product measure:
product_i x_i ,the product of the throughput of the individual connections, is also used as a measure of fairness. (in some contexts x_i is taken as the power of the i-th connection, and the product measure is referred to as network power.) the product measure is particularly sensitive to segregation; the product measure is zero if any connection receives zero throughput. in [MS91], it is shown that for a network with many connections and one shared gateway, the product measure is maximized when all connections receive the same throughput 🧚
Epsilon-fairness: a rate allocation is defined as epsilon-fair if
(min_i x_i) / (max_i x_i) >= 1 - epsilon.epsilon-fairness measures the worst-case ratio between any two throughput rates [ZKL04] 💜. epsilon-fairness is related to max-min fairness, defined later in this document 🎀
2.3.2. Metrics for Fairness between Flows with Different Resource Requirements ✨
okay so this part is about fairness between flows that are NOT the same, like different utility functions, different round-trip times, or different numbers of links on the path 🌸 Lots of these metrics can be written as solutions to utility maximization problems [K01], and the total utility covers both the fairness AND the throughput. the way that's connected is iconic 🦋
Max-min fairness: here the smallest throughput rate has to be as large as possible. once that's true, the next-smallest rate has to be as large as possible, and so on down the line. so max-min fairness gives absolute priority to the smallest flows 💖 (you can explain it with the progressive filling algorithm: every flow rate starts at zero, and they all grow at the same pace. each flow stops growing only when one or more links on its path hit link capacity. ngl this is so satisfying to picture 🌈)
Proportional fairness: now compare that to proportional fairness. an allocation, x, is called proportionally fair if, for any other feasible allocation x*, the sum of the proportional changes is zero or negative:
sum_i ( (x*_i - x_i)/x_i ) <= 0."This criterion favours smaller flows, but less emphatically than max-min fairness" [K01]. 💅 (If you speak in utility functions, proportional fairness comes from using logarithmic utility functions and maximizing the sum of the per-flow utility functions. for a fuller explanation, see [KMT98].)
Minimum potential delay fairness: this one has been shown to model TCP [KS03], and it's kind of a compromise between max-min fairness and proportional fairness 🎀 An allocation, x, has minimum potential delay fairness if:
sum_i (1/x_i)is smaller than for any other feasible allocation. basically, it minimizes the average download time if every flow is an equal-sized file. bestie that's such a clean way to think about it 🧚
In [CRM05], Colussi, et al. propose a new definition of TCP fairness: that "a set of TCP fair flows do not cause more congestion than a set of TCP flows would cause". congestion here is defined in terms of queueing delay, queueing delay variation, the congestion event rate [e.g., loss event rate], and the packet loss rate. 👑
Chiu and Tan in [CT06] argue that fairness should be redefined when you're studying traffic controls for inelastic traffic. they propose that inelastic flows use other traffic controls, such as admission control. not me being like "wait, a whole new definition?" 😭
The usefulness of flow-rate fairness has been challenged by Briscoe in [B07] and defended by Floyd and Allman in [FA08]. In [B07], Briscoe argues that flow-rate fairness should not be a desired goal, and that instead "we should judge fairness mechanisms on how they share out the 'cost' of each user's actions on others". Floyd and Allman in [FA08] argue that the current system, based on TCP congestion control and flow-rate fairness, has been useful in the real world. it poses minimal demands on network and economic infrastructure and lets users get a share of the network resources 🫶
2.3.3. Comments on Fairness
Trade-offs between fairness and throughput: the fairness measures above generally measure both fairness and throughput, giving different weights to each. Tang, et al. in [TWL06] also discuss potential trade-offs between fairness and throughput, for a framework where max-min fairness is defined as the most fair. [TWL06] shows that in some topologies, throughput is proportional to fairness, while in other topologies, throughput is inversely proportional to fairness. this is giving plot twist 🌪️✨
Fairness and the number of congested links: some of these fairness metrics are discussed in more detail in [F91]. there's not a clear consensus on fairness goals, especially for fairness between flows that cross different numbers of congested links [F91]. utility maximization gives one framework for describing this trade-off in fairness 🍓
Fairness and round-trip times: a bunch of new transport protocols have aimed for fairness between flows with different round-trip times [KHR02] [XHR04]. but the networking community doesn't agree on whether this goal is desirable, or on its implications and interactions with other metrics [FJ92] (Section 3.3). one common argument against it is that flows with long round-trip times consume more resources, and the previous paragraph covers this aspect. researchers have also noticed the difference between the RTT-unfairness of standard TCP and the greater RTT-unfairness of some proposed modifications to TCP [LLS05] 🦋
Fairness and packet size: one fairness issue is the relative fairness for flows with different packet sizes. lots of file transfer applications use the maximum packet size possible. low-bandwidth VoIP flows, on the other hand, are likely to send small packets, sending a new packet every 10 to 40 ms., to limit delay. so should a small-packet VoIP connection get the same sending rate in bytes per second as a large-packet TCP connection in the same environment? or should it get the same sending rate in packets per second? this has been discussed in more detail in [RFC3714], and [RFC4828] also describes the ways packet size can affect the packet drop rate a flow experiences 🎀
Convergence times: convergence times are about how long it takes for an existing flow and a newly starting one to converge to fairness. they're a special concern for environments with high-bandwidth long-delay flows. convergence times also cover how long it takes to reach fairness again after a sudden change, like a change in the network path, the competing cross-traffic, or the characteristics of a wireless link. as with fairness, convergence times can matter both between flows of the same protocol and between flows using different protocols [SLFK03] ⏱️
one metric for convergence times is the delta-fair convergence time. it's defined as the time taken for two flows with the same round-trip time to go from shares of 100/101-th and 1/101-th of the link bandwidth to close to fair sharing, with shares of (1+delta)/2 and (1-delta)/2 of the link bandwidth [BBFS01] 💜
a similar metric measures convergence time as the number of round-trip times for two flows to reach epsilon-fairness, when starting from a maximally-unfair state [ZKL04] 🌈
2.4. Robustness for Challenging Environments
while congestion control mechanisms are usually evaluated first over environments with static routing in a network of two-way point-to-point links, some environments bring up harder problems. these include corrupted packets, reordering, variable bandwidth, and mobility. they also bring new metrics to consider, like energy consumption 🌸
Robustness for challenging environments: robustness needs to be explored for paths with reordering, corruption, variable bandwidth, asymmetric routing, router configuration changes, mobility, and the like [GF04]. in general, the Internet architecture has valued robustness over efficiency, for example when there are trade-offs between robustness and the throughput, delay, and fairness metrics described above 🛡️
Energy consumption: in mobile environments, the energy consumption of the mobile end-node can be a key metric, and it's affected by the transport protocol [TM02] 🔋
The goodput ratio: for wireless networks, the goodput ratio can be a useful metric. it's defined as the useful data delivered to users as a fraction of the total amount of data transmitted on the network. a high goodput ratio indicates efficient use of the radio spectrum and lower interference with other users ✨
2.5. Robustness to Failures and to Misbehaving Users
one goal is for congestion control mechanisms to be robust to misbehaving users. examples are receivers that 'lie' to data senders about the congestion experienced along the path, or that otherwise try to bypass the sender's congestion control mechanisms [SCWA99]. 😤 another goal is for congestion control mechanisms to be as robust as possible to failures. examples include failures of routers when using explicit feedback to end-nodes, or failures of end-nodes to follow the prescribed protocols 🫠
2.6. Deployability
one metric for congestion control mechanisms is their deployability in the current Internet. metrics related to deployability include the ease of failure diagnosis and the overhead in terms of packet header size or added complexity at end-nodes or routers 🧩
one key aspect of deployability is the range of deployment needed for a new congestion control mechanism. consider the following possible deployment requirements:
- Only at the sender (e.g., NewReno in TCP [RFC3782]);
- Only at the receiver (e.g., delayed acknowledgements in TCP);
- Both the sender and receiver (e.g., Selective Acknowledgment (SACK) TCP [RFC2018]);
- At a single router (e.g., Random Early Detection (RED) [FJ93]);
- All of the routers along the end-to-end path;
- Both end-nodes and all routers along the path (e.g., Explicit Control Protocol (XCP) [KHR02]).
some congestion control mechanisms (e.g., XCP [KHR02], Quick-Start [RFC4782]) may also have deployment issues with IPsec, IP in IP, MPLS, other tunnels, or with non-router queues such as those in Ethernet switches 🌐
another deployability issue is the complexity of the code. how complex is the code required to implement the mechanism in software? is floating point math required? how much new state must be kept to implement the new mechanism, and who holds that state, the routers or the end-nodes? we note that we don't suggest these questions as ways to reduce the deployability metric to a single number. we suggest them as issues that could be considered when evaluating the deployability of a proposed congestion control mechanism 🤓
2.7. Metrics for Specific Types of Transport
in some cases, modified metrics are needed for evaluating transport protocols intended for quality-of-service (QoS)-enabled environments or for below-best-effort traffic [VKD02] [KK03] 💫
2.8. User-Based Metrics
an alternate approach that's been proposed for evaluating congestion control mechanisms is to evaluate them in terms of user metrics, such as user satisfaction, or in terms of application-specific utility functions. this approach would require defining a range of user metrics, or of application-specific utility functions, for the range of applications under consideration (e.g., FTP, HTTP, VoIP) 🥰
3. Metrics in the IP Performance Metrics (IPPM) Working Group ✨
okay so the IPPM Working Group [IPPM] was set up to define performance metrics that network operators, end users, or independent testing groups can use 💅. their list is a whole buffet: connectivity [RFC2678], delay and loss [RFC2679], [RFC2680], and [RFC2681], delay variation [RFC3393], loss patterns [RFC3357], packet reordering [RFC4737], bulk transfer capacity [RFC3148], and link capacity [RFC5136] 🎀. ngl the IPPM docs are so good bc they give concrete, well-defined metrics AND a methodology for measuring each one 🤓.
but here's the thing 🫶: the metrics in THIS document have a different purpose. we're talking about metrics used in analysis, simulations, and experiments to evaluate congestion control mechanisms 🔥. we're also discussing them in a general sense, not hunting for a specific concrete definition for each one. HOWEVER, there are lots of cases where IPPM definitions could be useful, like figuring out how to measure these metrics in simulations or testbeds, and having common definitions so everyone is talking about the same thing 🧚.
4. Comments on Methodology 🦋
the types of scenarios used to test specific metrics, and the range of parameters worth considering, will be covered in separate documents, e.g., alongside specific scenarios for evaluating congestion control mechanisms 📚.
it can matter a LOT to evaluate metrics across a wide range of environments, with different link bandwidths, congestion levels, and levels of statistical multiplexing 🌈. it's also important to evaluate congestion control mechanisms across a range of scenarios, including typical ranges of connection sizes and round-trip times [FK02] ✨. and it's useful to compare metrics for new or modified transport protocols against the current standards for TCP 💜.
as an example from the literature on evaluating transport protocols, Li, et al. in "Experimental Evaluation of TCP Protocols for High-Speed Networks" [LLS05] 👑 focus on TCP performance in high-speed networks. they look at metrics for aggregate throughput, loss rates, fairness (including fairness between flows with different round-trip times), response times (including convergence times), and incremental deployment 💖. more general methodology reading includes [J91] 📖, and papers that discuss the range of metrics for evaluating congestion control are in [MTZ04] 🍓.
5. Security Considerations 💀
Section 2.5 discusses the robustness of congestion control mechanisms to failures and to misbehaving users 😭. transport protocols also have other security concerns that have nothing to do with congestion control, and those aren't discussed in this document 🙈.
6. Acknowledgements 🫶
thank you to Lachlan Andrew, Mark Allman, Armando Caro, Dah Ming Chiu, Eric Coe, Dado Colussi, Wesley Eddy, Aaron Falk, Nelson Fonseca, Janardhan Iyengar, Doug Leith, Sara Landstrom, Tony Li, Saverio Mascolo, Sean Moore, Injong Rhee, David Ros, Juergen Schoenwaelder, Andras Veres, Michael Welzl, and Damon Wischik, plus everyone in the Transport Modeling Research Group for the feedback and contributions 🌸✨
7. Informative References ✨
okay bestie, this is the reading list 📚 every paper and RFC the author leaned on, and yes, I'm keeping every single citation exactly as written so you can go find them 🔍
[AEO03] M. Allman, W. Eddy, and S. Ostermann, Estimating Loss Rates
With TCP, ACM Performance Evaluation Review, 31(3),
December 2003.
[BB01] D. Bansal and H. Balakrishnan, Binomial Congestion Control
Algorithms, IEEE Infocom, April 2001.
[BBFS01] D. Bansal, H. Balakrishnan, S. Floyd, and S. Shenker,
Dynamic Behavior of Slowly-Responsive Congestion Control
Algorithms, SIGCOMM 2001.
[BJ81] K. Bharath-Kumar and J. Jeffrey, A New Approach to
Performance-Oriented Flow Control, IEEE Transactions on
Communications, Vol.29 N.4, April 1981.
[B07] B. Briscoe, "Flow Rate Fairness: Dismantling a Religion",
Computer Communications Review, V.37 N.2, April 2007.
[CRM05] D. Colussi, A New Approach to TCP-Fairness, Report C-2005-
49, University of Helsinki, Finland, 2005.
[CT06] D. Chiu and A. Tam, Redefining Fairness in the Study of
TCP-friendly Traffic Controls, Technical Report, 2006.
[DM06] N. Dukkipati and N. McKeown, Why Flow-Completion Time is
the Right Metric for Congestion Control, ACM SIGCOMM,
January 2006.
[F91] S. Floyd, Connections with Multiple Congested Gateways in
Packet-Switched Networks Part 1: One-way Traffic, Computer
Communication Review, Vol.21 No.5, October 1991, p. 30-47.
[FA08] S. Floyd and M. Allman, Comments on the Usefulness of
Simple Best-Effort Traffic, Work in Progress, January 2007.
[FF99] Floyd, S., and Fall, K., "Promoting the Use of End-to-End
Congestion Control in the Internet", IEEE/ACM Transactions
on Networking, August 1999.
[FHP00] S. Floyd, M. Handley, and J. Padhye, A Comparison of
Equation-Based and AIMD Congestion Control, May 2000. URL
http://www.icir.org/tfrc/.
[FHPW00] S. Floyd, M. Handley, J. Padhye, and J. Widmer, Equation-
Based Congestion Control for Unicast Applications, SIGCOMM
2000, August 2000.
[FJ92] S. Floyd and V. Jacobson, On Traffic Phase Effects in
Packet-Switched Gateways, Internetworking: Research and
Experience, V.3 N.3, September 1992, p.115-156.
[FJ93] S. Floyd and V. Jacobson, Random Early Detection gateways
for Congestion Avoidance, IEEE/ACM Transactions on
Networking, V.1 N.4, August 1993,
[FK02] S. Floyd and E. Kohler, Internet Research Needs Better
Models, Hotnets-I. October 2002.
[FK07] S. Floyd and E. Kohler, "Tools for the Evaluation of
Simulation and Testbed Scenarios", Work in Progress,
February 2008.
[GF04] A. Gurtov and S. Floyd, Modeling Wireless Links for
Transport Protocols, ACM CCR, 34(2):85-96, April 2004.
[HKLRX06] S. Ha, Y. Kim, L. Le, I. Rhee, and L. Xu, A Step Toward
Realistic Evaluation of High-speed TCP Protocols, technical
report, North Carolina State University, January 2006.
[HG86] E. Hahne and R. Gallager, Round Robin Scheduling for Fair
Flow Control in Data Communications Networks, IEEE
International Conference on Communications, June 1986.
[IPPM] IP Performance Metrics (IPPM) Working Group, URL
http://www.ietf.org/html.charters/ippm-charter.html.
[J91] R. Jain, The Art of Computer Systems Performance Analysis:
Techniques for Experimental Design, Measurement,
Simulation, and Modeling, John Wiley & Sons, 1991.
[JCH84] R. Jain, D.M. Chiu, and W. Hawe, A Quantitative Measure of
Fairness and Discrimination for Resource Allocation in
Shared Systems, DEC TR-301, Littleton, MA: Digital
Equipment Corporation, 1984.
[JWL04] C. Jin, D. Wei, and S. Low, FAST TCP: Motivation,
Architecture, Algorithms, Performance, IEEE INFOCOM, March
2004.
[K01] F. Kelly, Mathematical Modelling of the Internet,
"Mathematics Unlimited - 2001 and Beyond" (Editors B.
Engquist and W. Schmid), Springer-Verlag, Berlin, pp.
685-702, 2001.
[KHR02] D. Katabi, M. Handley, and C. Rohrs, Congestion Control for
High Bandwidth-Delay Product Networks, ACM Sigcomm, 2002.
[KK03] A. Kuzmanovic and E. W. Knightly, TCP-LP: A Distributed
Algorithm for Low Priority Data Transfer, IEEE INFOCOM
2003, April 2003.
[KMT98] F. Kelly, A. Maulloo and D. Tan, Rate Control in
Communication Networks: Shadow Prices, Proportional
Fairness and Stability. Journal of the Operational
Research Society 49, pp. 237-252, 1998.
[KS03] S. Kunniyur and R. Srikant, End-to-end Congestion Control
Schemes: Utility Functions, Random Losses and ECN Marks,
IEEE/ACM Transactions on Networking, 11(5):689-702, October
2003.
[LLS05] Y-T. Li, D. Leith, and R. Shorten, Experimental Evaluation
of TCP Protocols for High-Speed Networks, Hamilton
Institute, 2005. URL
http://www.hamilton.ie/net/eval/results_HI2005.pdf.
[MS91] D. Mitra and J. Seery, Dynamic Adaptive Windows for High
Speed Data Networks with Multiple Paths and Propagation
Delays, INFOCOM '91, pp 39-48.
[MTZ04] L. Mamatas, V. Tsaoussidis, and C. Zhang, Approaches to
Congestion Control in Packet Networks, 2004.
[RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP
Selective Acknowledgment Options", RFC 2018, October 1996.
[RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Packet Loss Metric for IPPM", RFC 2680, September 1999.
[RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring
Connectivity", RFC 2678, September 1999.
[RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Delay Metric for IPPM", RFC 2679, September 1999.
[RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
Delay Metric for IPPM", RFC 2681, September 1999.
[RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, RFC
2914, September 2000.
[RFC3148] Mathis, M. and M. Allman, "A Framework for Defining
Empirical Bulk Transfer Capacity Metrics", RFC 3148, July
2001.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of
Explicit Congestion Notification (ECN) to IP", RFC 3168,
September 2001.
[RFC3357] Koodli, R. and R. Ravikanth, "One-way Loss Pattern Sample
Metrics", RFC 3357, August 2002.
[RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation
Metric for IP Performance Metrics (IPPM)", RFC 3393,
November 2002.
[RFC3448] Handley, M., Floyd, S., Padhye, J., and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification", RFC
3448, January 2003.
[RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed.,
"RTP Control Protocol Extended Reports (RTCP XR)", RFC
3611, November 2003.
[RFC3714] Floyd, S., Ed., and J. Kempf, Ed., "IAB Concerns Regarding
Congestion Control for Voice Traffic in the Internet", RFC
3714, March 2004.
[RFC3782] Floyd, S., Henderson, T., and A. Gurtov, "The NewReno
Modification to TCP's Fast Recovery Algorithm", RFC 3782,
April 2004.
[RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, S.,
and J. Perser, "Packet Reordering Metrics", RFC 4737,
November 2006.
[RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick-
Start for TCP and IP", RFC 4782, January 2007.
[RFC4828] Floyd, S. and E. Kohler, "TCP Friendly Rate Control (TFRC):
The Small-Packet (SP) Variant", RFC 4828, April 2007.
[RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", RFC
5136, February 2008.
[RX05] I. Rhee and L. Xu, CUBIC: A New TCP-Friendly High-Speed TCP
Variant, PFLDnet 2005.
[SAF06] P. Sarolahti, M. Allman, and S. Floyd, Determining an
Appropriate Sending Rate Over an Underutilized Network
Path, Computer Networks, September 2006.
[SLFK03] R.N. Shorten, D.J. Leith, J. Foy, and R. Kilduff, Analysis
and Design of Congestion Control in Synchronised
Communication Networks. Proc. 12th Yale Workshop on
Adaptive & Learning Systems, May 2003.
[SCWA99] S. Savage, N. Cardwell, D. Wetherall, and T. Anderson, TCP
Congestion Control with a Misbehaving Receiver, ACM
Computer Communications Review, October 1999.
[TM02] V. Tsaoussidis and I. Matta, Open Issues of TCP for Mobile
Computing, Journal of Wireless Communications and Mobile
Computing: Special Issue on Reliable Transport Protocols
for Mobile Computing, February 2002.
[TWL06] A. Tang, J. Wang and S. H. Low. Counter-Intuitive
Throughput Behaviors in Networks Under End-to-End Control,
IEEE/ACM Transactions on Networking, 14(2):355-368, April
2006.
[WCL05] D. X. Wei, P. Cao and S. H. Low, Time for a TCP Benchmark
Suite?, Technical Report, Caltech CS, Stanford EAS,
Caltech, 2005.
[VKD02] A. Venkataramani, R. Kokku, and M. Dahlin, TCP Nice: A
Mechanism for Background Transfers, Fifth USENIX Symposium
on Operating System Design and Implementation (OSDI), 2002.
[XHR04] L. Xu, K. Harfoush, and I. Rhee, Binary Increase Congestion
Control for Fast, Long Distance Networks, Infocom 2004.
[YKL01] Y. Yang, M. Kim, and S. Lam, Transient Behaviors of TCP-
friendly Congestion Control Protocols, Infocom 2001.
[ZKL04] Y. Zhang, S.-R. Kang, and D. Loguinov, Delayed Stability
and Performance of Distributed Congestion Control, ACM
SIGCOMM, August 2004.Author's Address 🎀
the author's contact info, verbatim so you can reach out 💌
Sally Floyd
ICSI Center for Internet Research
1947 Center Street, Suite 600
Berkeley, CA 94704
USA
EMail: floyd@icir.orgFull Copyright Statement 🌈
the legal bit, kept word-for-word because it matters 💅
Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions
contained in BCP 78 and at http://www.rfc-editor.org/copyright.html,
and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.Intellectual Property ✨
okay so this part is all about who owns what 💅 The IETF doesn't take a side on whether any intellectual property rights (IPR) claimed around this technology are valid or how far they reach, and it doesn't say whether any license to those rights is available or not 🤷♀️ It also hasn't independently gone looking for any such rights, so don't assume it checked! 🙈 If you want the rules for rights in RFC documents, look at BCP 78 and BCP 79 📚
Want to see IPR disclosures? Copies of the ones made to the IETF Secretariat are available, along with any assurances of licenses that have been offered. You can also find the results of anyone trying to get a general license or permission for implementers or users to use the proprietary rights. Everything lives in the IETF online IPR repository at http://www.ietf.org/ipr 🗂️✨
And the IETF is literally asking everyone to speak up 📣 If you know about any copyrights, patents, patent applications, or other proprietary rights that might cover technology you'd need to implement this standard, tell the IETF! Send it to ietf-ipr@ietf.org 💖 Bestie, that's the whole vibe: be transparent, be helpful, and keep the receipts 🧾🦋
📦 where this page comes from
- ietf: rfc5166 2008-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms