RFC 2884

Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks

📅 published July 2000, from ietf

RFC 2884 🎀 Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks

okay bestie, here's the tea 🍵 RFC 2884 is titled Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks. It was written by J. Hadi Salim (Nortel Networks) and U. Ahmed (Carleton University) in July 2000.

It's Informational, so it's giving "here's some info" and not "this is a standard" 💅. It doesn't specify an Internet standard of any kind. The header doesn't list anything it obsoletes or updates, so it's a solo vibe 🦋

Status of This Memo ✨

This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.

Copyright (C) The Internet Society (2000). All Rights Reserved.

Abstract 💜

Okay so this memo is a performance study of ECN in the TCP/IP protocol, tested on the Linux Operating System with their own implementation 🔥 ECN is an end-to-end congestion avoidance mechanism proposed by [6] and brought into RFC 2481[7]. They looked at how ECN behaves for both bulk transfers and transactional transfers.

The results are iconic ✨ For bulk transfers there's an improvement in throughput over NON ECN (that's TCP using any of Reno, SACK/FACK or NewReno congestion control). For transactional transfers the improvement is substantial 🙌

There's a more complete pdf version of this document at:

http://www7.nortel.com:8080/CTL/ecnperf.pdf

This memo's current revision is missing a lot of the visuals and experimental results that are in the pdf version. Go check it out if you want the full picture 👀

1. Introduction 🦋

In current IP networks, congestion management is left to the protocols running on top of IP. When an IP router gets congested, it just drops packets 💀 TCP is the dominant transport protocol today [26].

TCP figures out there's congestion by detecting packet drops (RFC 2581). Congestion control algorithms [11] [15] [21] then kick in to ease things up. TCP starts out sending at a higher rate (that's slow start) until it notices a packet loss. A loss gets inferred when it receives 3 duplicate ACKs or when a timeout happens. After that, the sending TCP moves into a congestion avoidance state, where it carefully probes the network by sending at a slower rate. The rate goes back up until another packet loss is detected 🌈

Traditionally, a router reacts to congestion by dropping a packet when there's no buffer space left. This is called Tail Drop. It has a bunch of drawbacks (outlined in Section 2), and those drawbacks, plus the limits of end-to-end congestion control, have people excited about smarter congestion control mechanisms in routers ✨

One such mechanism is Random Early Detection (RED) [9]. RED detects incipient congestion and implicitly signals the oversubscribing flow to slow down by dropping its packets. A RED-enabled router spots congestion before the buffer overflows, based on a running average queue size. It drops packets probabilistically before the queue actually fills up. The chance of dropping a newly arriving packet goes up as the average queue size climbs above a low water mark minth, heading toward a higher water mark maxth. Once the average queue size goes past maxth, every arriving packet gets dropped 🙈

Here's the extension to RED: instead of dropping packets, mark the IP header (when the average queue size is between minth and maxth). Above maxth, arriving packets still get dropped like before. Cooperating end systems use the mark as a signal that the network is congested and slow down 🫶 This is known as Explicit Congestion Notification (ECN).

In this paper, they study an ECN implementation on Linux for both the router and the end systems in a live network 💖 Here's how the memo is organized:

  • Section 2 gives an overview of queue management in routers.
  • Section 3 gives an overview of ECN and the changes required at the router and the end hosts to support ECN.
  • Section 4 defines the experimental testbed and the terminologies used throughout this memo.
  • Section 5 introduces the experiments that are carried out, outlines the results and presents an analysis of the results obtained.
  • Section 6 concludes the paper.

2. Queue Management in routers 🎀

TCP's congestion control and avoidance algorithms are necessary and powerful, but they aren't enough to give good service in all circumstances. They treat the network as a black box 📦 Some sort of control is required from the routers to complement the end system congestion control mechanisms. More detailed analysis is in [19].

Queue management algorithms traditionally manage the length of packet queues in the router by dropping packets only when the buffer overflows. Each queue has a configured maximum length. The router accepts packets until this maximum size is exceeded, at which point it drops incoming packets. New packets get accepted again when buffer space allows. This is called Tail Drop 💅

This method has served the Internet well for years, but it has several drawbacks. Since all arriving packets (from all flows) get dropped when the buffer overflows, it interacts badly with TCP's congestion control mechanism 😭 A cycle forms: a burst of drops happens after the maximum queue size is exceeded, then there's a period of underutilization at the router as end systems back off. Then end systems increase their windows at the same time, up to the point where another burst of drops happens. This is called Global Synchronization. It leads to poor link utilization and lower overall throughput [19]

Another problem with Tail Drop is that a single connection or a few flows could monopolize the queue space in some circumstances. This causes a lock out phenomenon, leading to synchronization or other timing effects [19]. Lastly, one of the major drawbacks of Tail Drop is that queues stay full for long periods of time. One of the major goals of queue management is to reduce the steady state queue size [19].

Other queue management techniques include random drop on full and drop front on full [13].

2.1. Active Queue Management 🌸

Active queue management mechanisms detect congestion before the queue overflows, and they give an indication of that congestion to the end nodes [7]. With this approach, TCP doesn't have to rely only on buffer overflow as the sign of congestion, because notification happens before serious congestion occurs. One such active management technique is RED.

2.1.1. Random Early Detection 🧚

Random Early Detection (RED) [9] is a congestion avoidance mechanism implemented in routers, and it works on the basis of active queue management. RED addresses the shortcomings of Tail Drop 💖 A RED router signals incipient congestion to TCP by dropping packets probabilistically, before the queue runs out of buffer space.

The drop probability depends on a running average queue size, so it avoids any bias against bursty traffic. A RED router randomly drops arriving packets. The result is that the probability of dropping a packet from a particular flow is roughly proportional to that flow's share of bandwidth. So if the sender is using relatively more bandwidth, it gets penalized by having more of its packets dropped 👑

RED works by maintaining two thresholds, a minimum (minth) and a maximum (maxth). It drops a packet probabilistically if and only if the average queue size lies between minth and maxth. If the average queue size is above the maximum threshold, the arriving packet is always dropped. When the average queue size is between the minimum and maximum thresholds, each arriving packet gets dropped with probability pa, where pa is a function of the average queue size. As the average queue length varies between minth and maxth, pa increases linearly toward a configured maximum drop probability, maxp. Beyond maxth, the drop probability is 100% 💀

Dropping packets this way makes sure that when some subset of the source TCP packets get dropped, they invoke congestion avoidance algorithms that ease the congestion at the gateway. Since the dropping is distributed across flows, the problem of global synchronization is avoided 🙌

3. Explicit Congestion Notification 💜

Explicit Congestion Notification is an extension proposed to RED. It marks a packet instead of dropping it when the average queue size is between minth and maxth [7]. Since ECN marks packets before congestion actually occurs, it's useful for protocols like TCP that are sensitive to even a single packet loss 🥺

When the TCP receiver gets a congestion marked packet, it informs the sender (in the subsequent ACK) about incipient congestion. That in turn triggers the congestion avoidance algorithm at the sender 🔥

ECN requires support from both the router and the end hosts. That means the end hosts' TCP stack needs to be modified 🛠️ Packets from flows that aren't ECN capable will keep getting dropped by RED, same as before ECN existed 🙈

3.1. Changes at the router 🦋

Router side support for ECN can be added by modifying current RED implementations. For packets from ECN capable hosts, the router marks the packets instead of dropping them (if the average queue size is between minth and maxth) ✨

It's necessary that the router identifies that a packet is ECN capable, and it should only mark packets from ECN capable hosts. This uses two bits in the IP header 🎀

The ECN Capable Transport (ECT) bit is set by the sender end system if both end systems are ECN capable. For a unicast transport, it's only set if both end systems are ECN-capable. In TCP, this gets confirmed in the pre-negotiation during the connection setup phase (explained in Section 3.2) 🤝

Packets encountering congestion get marked by the router using the Congestion Experienced (CE) bit (if the average queue size is between minth and maxth) on their way from the sender end system to the receiver end system. The marking probability is proportional to the average queue size, following the procedure used in RED (RFC2309) routers 🌈

Bits 10 and 11 in the IPV6 header are proposed respectively for the ECT and CE bits. Bits 6 and 7 of the IPV4 header DSCP field are also specified for experimental purposes, for the ECT and CE bits respectively 🧚

Part 1 of 5 is done ✨ More parts are coming, so don't go anywhere 💜🫶

3.2. Changes at the TCP Host side 🎀

okay bestie, so the ECN proposal for TCP adds two new flags to the reserved field of the TCP header 💖. Bit 9 of that reserved field is the ECN-Echo (ECE) flag, and Bit 8 is the Congestion Window Reduced (CWR) flag 🦋. These two bits do double duty ✨: they handle the setup phase where sender and receiver negotiate whether they both want to use ECN, and they also handle what happens after that, when congestion shows up on the network during an established connection 😭.

There are basically two big changes to the end system and one extension to a router running RED. Let's go 💅

  1. Connection setup. The source and destination TCPs need to swap info about whether they want to use ECN and whether they're even capable of it 🫶. The sender sets both the ECN-Echo flag and the CWR flag in the SYN packet at the start of the connection 🌸. When the receiver gets that SYN, it sets the ECN-Echo flag in the SYN-ACK response 🧚. Once that handshake is done, the sender starts setting the ECT bit in the IP header of data packets for that flow, which tells the network "hey, I'm capable and willing to do ECN" 💜. The ECT bit goes on all packets except pure ACKs 🙈.
  2. Telling the sender about congestion. When a router's active queue management decides to drop or mark a packet, it checks the IP-ECT bit in that packet's header 👑. If the IP-ECT bit is set, the router sets the CE bit in the IP header instead of dropping it 🔥. When that marked packet reaches the receiver, the receiver sets the ECN-Echo flag (in the TCP header) on its next outgoing ACK for the flow 🌈. It keeps doing that on subsequent ACKs until the sender says, "okay, I've responded to your congestion notification" 🍓.
  3. Sender reacts. When the sender gets that ACK, it triggers its congestion avoidance algorithm ✨. It halves its congestion window, cwnd, and updates its congestion window threshold, ssthresh 💅. After those steps, the sender sets the CWR bit on its next outgoing data packet to tell the receiver it has reacted to the congestion notification 🫶. The receiver then stops sending ECE to the sender, as long as there's no new congestion in the network 🦋.

Here's the tea 🍵: when the sender gets an ACK with the ECN-Echo flag set, its response is basically the same as Fast Retransmit/Recovery in NON-ECN-capable TCP when there's a congestion loss 😭. In other words, the sender halves cwnd and reduces the slow start threshold, ssthresh 💖. Fast Retransmit/Recovery is still available to ECN-capable stacks for responding to three duplicate acknowledgments 🎀.

4. Experimental setup 🧪

Okay, so for testing, the team added ECN to the Linux TCP/IP stack 🐧. The kernel versions were 2.0.32, 2.2.5, and 2.3.43, and there were also earlier revisions of 2.3 that got tested too 💜. The 2.0.32 implementation follows RFC 2481 [7] for the end systems only 🌸. For the 2.1, 2.2, and 2.3 cases, they also modified the router portion as well as the end system so it conforms to the RFC ✨. An outdated version of the 2.0 code is available at [18] 🍓.

Quick aside, because this part matters 👀: Linux 2.0.32 runs TCP Reno congestion control, while kernels >= 2.2.0 default to New Reno, but they'll go with a SACK/FACK combo when the remote end understands SACK 🔥. The first tests used the 2.0 kernel at the end system and 2.1 (pre 2.2) for the router 🦋. Most of the results here come from the 2.0 tests 💅. They repeated the tests on a different testbed, moving from Pentium to Pentium-II class machines, with faster machines for the 2.2 and 2.3 kernels 🫶. So the comparisons between 2.0 and 2.2/3 aren't relative to each other 🙈.

They've also updated this memo to reflect the tests against SACK and New Reno 💖.

4.1. Testbed setup 🌈

                                             -----      ----
                                            | ECN |    | ECN |
                                            | ON  |    | OFF |
          data direction ---->>              -----      ----
                                              |          |
      server                                  |          |
       ----        ------        ------       |          |
      |    |      |  R1  |      |  R2  |      |          |
      |    | -----|      | ---- |      | ----------------------
       ----        ------ ^      ------             |
                          ^                         |
                          |                        -----
      congestion point ___|                       |  C  |
                                                  |     |
                                                   -----

The figure above shows the test setup 👆.

Every physical link is 10Mbps ethernet 🔌. Using Class Based Queuing (CBQ) [22], packets from the data server are constricted to a 1.5Mbps pipe at router R1 🎀. Data is always retrieved from the server toward the clients labelled "ECN ON", "ECN OFF", and "C" 💌. Since the pipe from the server is 10Mbps, this creates congestion at the exit from R1 toward the clients for competing flows 💥. The "ECN ON" and "ECN OFF" machines run the same version of Linux and have exactly the same hardware configuration 👯. The server is always ECN capable, and it can handle NON ECN flows too, using the standard congestion algorithms 🫶. The machine labeled "C" is used to create congestion in the network 😈. Router R2 acts as a path-delay controller, so we can adjust the RTT the clients see 🕰️. Router R1 has RED implemented and can support ECN flows 🦋. The path-delay router is a PC running the Nistnet [16] package on Linux 🐧. The link latency for the experiments was set to 20 milliseconds ⏱️.

4.2. Validating the Implementation 🔍

We spent time validating that the implementation conformed to the specification in RFC 2481 📜. To do this, they modified the popular tcpdump sniffer [24] so it shows the packets being marked 👀. They visually inspected tcpdump traces to check conformance to the RFC across lots of different scenarios 🕵️‍♀️. They also modified tcptrace [25] to plot the marked packets for visualization and analysis 📈.

Both tcpdump and tcptrace showed that the implementation was conformant to the RFC 🎉.

4.3. Terminology used 📚

This section covers the background terminology used in the next few sections 🧠.

  • Congesting flows: TCP flows started in the background to create congestion from R1 toward R2 💥. The laptop labeled "C" introduces these congesting flows. Note that "C", like the other clients, retrieves data from the server 💻.
  • Low, Moderate and High congestion: For low congestion, we start two congesting flows in the background. For moderate congestion, we start five. For high congestion, we start ten 📶.
  • Competing flows: These are the flows we actually care about 👑. They're either ECN TCP flows from/to "ECN ON" or NON ECN TCP flows from/to "ECN OFF" 🫶.
  • Maximum drop rate: The RED parameter that sets the maximum probability of a packet being marked at the router 🎯. This corresponds to maxp, as explained in Section 2.1 🦋.

Our tests were repeated for different congestion levels and different maximum drop rates 🔁. The results are in the sections that follow 🌸.

  • Low, Medium and High drop probability: Low probability means a drop probability maxp of 0.02, medium is 0.2, and high is 0.5 🌡️. They also experimented with drop probabilities of 0.05, 0.1, and 0.3 🍓.
  • Goodput: The effective data rate as the user observes it 📦. For example, if 4 data packets were transmitted and 2 of them were retransmitted, the efficiency is 50%, and the resulting goodput is 2*packet size/time taken to transmit 🧮.
  • RED Region: When the router's average queue size is between minth and maxth, we say the router is operating in the RED region 🎢.

4.4. RED parameter selection 🎛️

In early testing, they noticed that as the number of congesting flows went up, the RED queue degenerated into a simple Tail Drop queue 😭. In other words, the average queue exceeded the maximum threshold most of the time 💀. This phenomenon has also been observed by [5], which proposes a dynamic solution to fix it by adjusting the packet dropping probability "maxp" based on the past history of the average queue size 🧠. So it was necessary that the router operate in the RED region during the experiments, meaning the average queue had to stay between minth and maxth 🎯. If it didn't, the queue would act like a Tail Drop queue, and the advantages of ECN would diminish 📉. The goal was to validate ECN's benefits when used with RED at the router 💖. To make sure they were operating in the RED region, they monitored the average and actual queue sizes under low, moderate, and high congestion, then fine-tuned the RED parameters so the average queue hovered around the RED region before running the real experiment 🎛️. So the results weren't skewed by operating in the wrong RED region 🙌.

5. The Experiments 🧪✨

We start by making sure the background flows don't bias our results, by computing the fairness index [12] in Section 5.1 ⚖️. Next we run the bulk transfer experiments and present the results and analysis in Section 5.2 📦. Section 5.3 presents the results and analysis for transactional transfers 🛒. More details on the experimental results can be found in [27] 📚.

5.1. Fairness ⚖️

okay so bestie, before we trust ANY of our numbers we had to check that picking the background flows wasn't quietly rigging the results 🙈 so we first ran tests with both ECN and NON ECN flows as the background. we repeated the experiments for different drop probabilities and calculated the fairness index [12]. 💅 and here's the lowkey iconic part: when there were an equal number of ECN and NON ECN flows, the number of packets dropped for the NON ECN flows was equal to the number of packets marked for the ECN flows. so RED was fair to both kinds of flows 🦋

Fairness index: this is a performance metric from [12]. Jain [12] says the network is a multi-user system and makes a metric to see how fairly each user gets treated. fairness is a function of how much throughput varies across users. for a set of user throughputs (x1, x2...xn), the fairness index is:

f(x1,x2,.....,xn) = square((sum[i=1..n]xi))/(n*sum[i=1..n]square(xi))

the fairness index always lands between 0 and 1. a value of 1 means every flow got exactly the same throughput ✨ each test ran 10 times so we could trust our results 🎀 and to compute the fairness index we used FTP to generate traffic 🧚

Experiment details: at time t = 0 we start 2 NON ECN FTP sessions in the background to create congestion 🌸 then at t=20 seconds we start two competing flows. we note the throughput of every flow in the network and calculate the fairness index. we ran this for various maximum drop probabilities and various congestion levels. then we repeated the whole thing with the background flows as ECN 💜 the fairness index stayed pretty constant in both cases, so the nature of the background flows wasn't biasing anything. no bias with ECN, no bias with NON ECN, periodt 💅

Here's the table, in all its glory:

Max     Fairness                Fairness
Drop    With BG                 With BG
Prob    flows ECN               flows NON ECN

0.02    0.996888                0.991946
0.05    0.995987                0.988286
0.1     0.985403                0.989726
0.2     0.979368                0.983342
✨ since the nature of the background flows doesn't change the results, we're using NON ECN background flows for the rest of the experiments. easy decision, no drama 🫶

5.2. Bulk transfers 🚚

the metric we picked for bulk transfers is end user throughput 💖

Experiment Details: every TCP flow here is RENO TCP 🦋 for low congestion we start 2 FTP flows in the background at time 0. after about 20 seconds we start the competing flows: one data transfer to the ECN machine and a second to the NON ECN machine. the file size is 20MB. for moderate congestion we start 5 FTP flows in the background, and for high congestion we start 10 🔥 we repeat the experiments for various maximum drop rates, each one repeated for a number of sets.

Observation and Analysis:

We make three key observations:

  1. As the congestion level increases, the relative advantage for ECN increases but the absolute advantage decreases (expected, since there are more flows competing for the same link resource). ECN still does better than NON ECN even under high congestion. Infering a sample from the collected results: at maximum drop probability of 0.1, for example, the relative advantage of ECN increases from 23% to 50% as the congestion level increases from low to high. 📈
  2. Maintaining congestion levels and varying the maximum drop probability (MDP) reveals that the relative advantage of ECN increases with increasing MDP. As an example, for the case of high congestion as we vary the drop probability from 0.02 to 0.5, the relative advantage of ECN increases from 10% to 60%. 🌈
  3. There were hardly any retransmissions for ECN flows (except the occasional packet drop in a minority of the tests for the case of high congestion and low maximum drop probability). 🍓

okay so we analysed tcpdump traces for NON ECN with the help of tcptrace 🕵️‍♀️ and observed there were hardly any retransmits due to timeouts. (retransmits due to timeouts are inferred by counting the 3 DUPACKS retransmits and subtracting them from the total recorded number of retransmits.) this means that over a long period of time (as with long bulk transfers), the data-driven loss recovery of the Fast Retransmit/Recovery algorithm is very effective 👑 the algorithm for ECN on congestion notification from ECE is the same as the one for a Fast Retransmit for NON ECN. since both are operating in the RED region, ECN barely gets any advantage over NON ECN from the signaling (packet drop vs. marking) 🤷‍♀️

that said, the results make it clear that ECN flows benefit in bulk transfers 💅 we think the main advantage is that less time gets spent recovering (whereas NON ECN spends time retransmitting), and timeouts are avoided altogether ⭐ [23] has shown that even with RED deployed, TCP RENO could suffer from multiple packet drops within the same window of data, which is likely to lead to multiple congestion reactions or timeouts (and ECN alleviates these problems). however, while TCP Reno has performance problems with multiple packets dropped in a window of data, New Reno and SACK have no such problems 🙌

so for scenarios with very high congestion, the advantages of ECN for TCP Reno flows could be more dramatic than the advantages of ECN for NewReno or SACK flows. an important observation from our results is that we do not notice multiple drops within a single window of data. so we'd expect our results aren't heavily influenced by Reno's performance problems with multiple packets dropped from a window of data 🫠 we repeated these tests with ECN-patched newer Linux kernels. as mentioned earlier, these kernels would use a SACK/FACK combo with a fallback to New Reno. SACK can be selectively turned off (defaulting to New Reno). our results indicate ECN still improves performance for the bulk transfers 🎉 more results are available in the pdf version[27]. as in 1) above, maintaining a maximum drop probability of 0.1 and increasing the congestion level, ECN-SACK improves performance from about 5% at low congestion to about 15% at high congestion. in the scenario where high congestion is maintained and the maximum drop probability moves from 0.02 to 0.5, the relative advantage of ECN-SACK improves from 10% to 40%. these numbers are lower than the ones Reno showed, but they still reflect the improvement ECN offers even alongside robust recovery mechanisms like SACK 💜

5.3. Transactional transfers

okay bestie, now we're doing transactional transfers 🎀. The idea is super simple: send a tiny request, wait for the server to answer, and only then send the next request 💅. To generate this traffic we use Netperf [17] with the CRR (Connect Request Response) option ✨. Say we're grabbing a small file, like 5 - 20 KB. We send a small request, the server sends the file back, and the transaction is complete once we've received the whole file 📦. To feel confident in our results we run the simulation for about one hour 🕐, and each test has a few thousand of these request and response exchanges going on.

Netperf-CRR isn't exactly HTTP 1.0 traffic, where several concurrent sessions get opened 🙈. It is still a really close approximation, though 🫶. Netperf-CRR waits for one connection to finish before it opens the next one (0 think time), so that single connection can be seen as the slowest response in the group of concurrent sessions (in HTTP) 🐢.

We picked the transactional data sizes based on [2], which says the average web transaction was around 8 - 10 KB 🌸. The smaller (5KB) size was chosen to guestimate the size of transactional processing that may become prevalent with policy management schemes in the diffserv [4] context 🔮. With Netperf we can start these transactional transfers and keep them going for a variable length of time ⏱️. The main metric we care about here is the transaction rate, which Netperf records for us 📊.

  • Define Transaction rate as: The number of requests and complete responses for a particular requested size that we are able to do per second. For example if our request is of 1KB and the response is 5KB then we define the transaction rate as the number of such complete transactions that we can accomplish per second. 💖

Experiment Details: Just like with bulk transfers, we start the background FTP flows at time 0 to add congestion to the network 🚦. About 20 seconds later we kick off the transactional transfers and run each test for three minutes 🏁. We record the transactions per second that complete ✅. We repeat the test for about an hour, then plot the various transactions per second, averaged over the runs 📈. The whole experiment gets repeated for various maximum drop probabilities, file sizes, and levels of congestion 🌈.

Observation and Analysis

okay, the tea ☕ There are three key observations:

  1. As congestion increases (with fixed drop probability) the relative advantage for ECN increases (again the absolute advantage does not increase since more flows are sharing the same bandwidth). For example, from the results, if we consider the 5KB transactional flow, as we increase the congestion from medium congestion (5 congesting flows) to high congestion (10 congesting flows) for a maximum drop probability of 0.1 the relative gain for ECN increases from 42% to 62% 📈🔥.
  2. Keeping the congestion level the same while adjusting the maximum drop probability shows that the relative advantage for ECN flows increases. From the case of high congestion for the 5KB flow we observe that the number of transactions per second increases from 0.8 to 2.2, which corresponds to an increase in relative gain for ECN of 20% to 140% 🤯.
  3. As the transactional data size increases, ECN's advantage diminishes because the probability of recovering from a Fast Retransmit increases for NON ECN. ECN, therefore, has a huge advantage as the transactional data size gets smaller, as is observed in the results 🧚. This can be explained by looking at TCP recovery mechanisms. NON ECN in the short flows depends, for recovery, on congestion signaling via receiving 3 duplicate ACKs, or worse by a retransmit timer expiration, whereas ECN depends mostly on the TCP-ECE flag 🚩. This is by design in our experimental setup. [3] shows that most of the TCP loss recovery in fact happens in timeouts for short flows 😭. The effectiveness of the Fast Retransmit/Recovery algorithm is limited by the fact that there might not be enough data in the pipe to elicit 3 duplicate ACKs. TCP RENO needs at least 4 outstanding packets to recover from losses without going into a timeout. For 5KB (4 packets for MTU of 1500Bytes) a NON ECN flow will always have to wait for a retransmit timeout if any of its packets are lost. ( This timeout could only have been avoided if the flow had used an initial window of four packets, and the first of the four packets was the packet dropped). We repeated these experiments with the kernels implementing SACK/FACK and New Reno algorithms. Our observation was that there was hardly any difference with what we saw with Reno. For example, in the case of SACK-ECN enabling, maintaining the maximum drop probability to 0.1 and increasing the congestion level for the 5KB transaction, we noticed that the relative gain for the ECN enabled flows increases from 47-80%. If we maintain the congestion level for the 5KB transactions and increase the maximum drop probabilities instead, we notice that SACKs performance increases from 15%-120%. It is fair to comment that the difference in the testbeds (different machines, same topology) might have contributed to the results; however, it is worth noting that the relative advantage of the SACK-ECN is obvious 💜.
⚠️ Heads up: the numbers in this section come from one specific testbed, so the difference in machines might have played a part. The relative advantage of SACK-ECN is still obvious though 👑.

6. Conclusion 👑

ECN enhancements improve on both bulk and transactional TCP traffic 🌟. The improvement is more obvious in short transactional type of flows (popularly referred to as mice) 🐭.

  • Because fewer retransmits happen with ECN, there is less traffic on the network 🚀. Although the relative amount of data retransmitted in our case is small, the effect could be higher when there are more contributing end systems. The absence of retransmits also implies an improvement in the goodput. This becomes very important for scenarios where bandwidth is expensive, such as in low bandwidth links 💸. This implies also that ECN lends itself well to applications that require reliability but would prefer to avoid unnecessary retransmissions 🫶.
  • The fact that ECN avoids timeouts by getting faster notification (as opposed to traditional packet dropping inference from 3 duplicate ACKs or, even worse, timeouts) implies less time is spent during error recovery. This also improves goodput ⚡.
  • ECN could be used to help in service differentiation, where the end user is able to "probe" for their target rate faster 🎯. Assured forwarding [1] in the diffserv working group at the IETF proposes using RED with varying drop probabilities as a service differentiation mechanism 🦋. It is possible that multiple packets within a single window in TCP RENO could be dropped even in the presence of RED, likely leading into timeouts [23]. ECN end systems ignore multiple notifications, which helps counter this scenario and results in improved goodput 💅. The ECN end system also ends up probing the network faster (to reach an optimal bandwidth). [23] also notes that RENO is the most widely deployed TCP implementation today 📦.

It is clear that the advent of policy management schemes introduces new requirements for transactional type of applications, which constitute a very short query and a response in the order of a few packets 🧚. ECN provides advantages to transactional traffic, as we have shown in the experiments ✨.

7. Acknowledgements 🫶

We would like to thank Alan Chapman, Ioannis Lambadaris, Thomas Kunz, Biswajit Nandy, Nabil Seddigh, Sally Floyd, and Rupinder Makkar for their helpful feedback and valuable suggestions 💖.

8. Security Considerations 🔒

Security considerations are as discussed in section 9 of RFC 2481 🚨.

9. References 📚

[1] Heinanen, J., Finland, T., Baker, F., Weiss, W. and J. Wroclawski, "Assured Forwarding PHB Group", RFC 2597, June 1999.

[2] B.A. Mat. "An empirical model of HTTP network traffic." In proceedings INFOCOMM'97.

[3] Balakrishnan H., Padmanabhan V., Seshan S., Stemn M. and Randy H. Katz, "TCP Behavior of a busy Internet Server: Analysis and Improvements", Proceedings of IEEE Infocom, San Francisco, CA, USA, March '98 http://nms.lcs.mit.edu/~hari/papers/infocom98.ps.gz

[4] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z. and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998.

[5] W. Feng, D. Kandlur, D. Saha, K. Shin, "Techniques for Eliminating Packet Loss in Congested TCP/IP Networks", U. Michigan CSE-TR-349-97, November 1997.

[6] S. Floyd. "TCP and Explicit Congestion Notification." ACM Computer Communications Review, 24, October 1994.

[7] Ramakrishnan, K. and S. Floyd, "A Proposal to add Explicit Congestion Notification (ECN) to IP", RFC 2481, January 1999.

[8] Kevin Fall, Sally Floyd, "Comparisons of Tahoe, RENO and Sack TCP", Computer Communications Review, V. 26 N. 3, July 1996, pp. 5-21

[9] S. Floyd and V. Jacobson. "Random Early Detection Gateways for Congestion Avoidance". IEEE/ACM Transactions on Networking, 3(1), August 1993.

[10] E. Hashem. "Analysis of random drop for gateway congestion control." Rep. Lcs tr-465, Lav. Fot Comput. Sci., M.I.T., 1989.

[11] V. Jacobson. "Congestion Avoidance and Control." In Proceedings of SIGCOMM '88, Stanford, CA, August 1988.

[12] Raj Jain, "The art of computer systems performance analysis", John Wiley and sons QA76.9.E94J32, 1991.

[13] T. V. Lakshman, Arnie Neidhardt, Teunis Ott, "The Drop From Front Strategy in TCP Over ATM and Its Interworking with Other Control Features", Infocom 96, MA28.1.

[14] P. Mishra and H. Kanakia. "A hop by hop rate based congestion control scheme." Proc. SIGCOMM '92, pp. 112-123, August 1992.

[15] Floyd, S. and T. Henderson, "The NewReno Modification to TCP's Fast Recovery Algorithm", RFC 2582, April 1999.

[16] The NIST Network Emulation Tool http://www.antd.nist.gov/itg/nistnet/

[17] The network performance tool http://www.netperf.org/netperf/NetperfPage.html

[18] ftp://ftp.ee.lbl.gov/ECN/ECN-package.tgz

[19] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, S., Wroclawski, J. and L. Zhang, "Recommendations on Queue Management and Congestion Avoidance in the Internet", RFC 2309, April 1998.

[20] K. K. Ramakrishnan and R. Jain. "A Binary feedback scheme for congestion avoidance in computer networks." ACM Trans. Comput. Syst., 8(2):158-181, 1990.

[21] Mathis, M., Mahdavi, J., Floyd, S. and A. Romanow, "TCP Selective Acknowledgement Options", RFC 2018, October 1996.

[22] S. Floyd and V. Jacobson, "Link sharing and Resource Management Models for packet Networks", IEEE/ACM Transactions on Networking, Vol. 3 No.4, August 1995.

[23] Prasad Bagal, Shivkumar Kalyanaraman, Bob Packer, "Comparative study of RED, ECN and TCP Rate Control". http://www.packeteer.com/technology/Pdf/packeteer-final.pdf

[24] tcpdump, the protocol packet capture & dumper program. ftp://ftp.ee.lbl.gov/tcpdump.tar.Z

[25] TCP dump file analysis tool: http://jarok.cs.ohiou.edu/software/tcptrace/tcptrace.html

[26] Thompson K., Miller, G.J., Wilder R., "Wide-Area Internet Traffic Patterns and Characteristics". IEEE Networks Magazine, November/December 1997.

[27] http://www7.nortel.com:8080/CTL/ecnperf.pdf

10. Authors' Addresses 💌

okay bestie, the people who wrote this are Jamal Hadi Salim from Nortel Networks 🍓 in Ottawa, ON, Canada. You can reach him at hadi@nortelnetworks.com 📬

and Uvaiz Ahmed from Carleton University's Dept. of Systems and Computer Engineering, also in Ottawa, Canada 🎀. His email is ahmed@sce.carleton.ca 💜

the address block for Jamal is:

Jamal Hadi Salim
Nortel Networks
3500 Carling Ave
Ottawa, ON, K2H 8E9
Canada

EMail: hadi@nortelnetworks.com

and for Uvaiz:

Uvaiz Ahmed
Dept. of Systems and Computer Engineering
Carleton University
Ottawa
Canada

EMail: ahmed@sce.carleton.ca

okay so this part is the legal bit, but it's actually kind of iconic 💅 Copyright (C) The Internet Society (2000). All Rights Reserved. ✨

the gist: you can copy this document and share it, and you can make derivative works that comment on or explain it or help with implementing it, as long as the copyright notice and this paragraph stay on every copy and derivative work 🦋 ngl the document itself can't be modified in any way, like you can't remove the copyright notice or the references to the Internet Society or other Internet organizations 🙈 the exceptions are: if you're developing Internet standards (then the Internet Standards process procedures for copyrights have to be followed), or if you're translating it into a language other than English 🌸

the permissions here are perpetual, so the Internet Society (or its successors or assigns) can't take them back 🫶

and the big disclaimer, in all caps because it's serious: this document and the info in it is provided on an "AS IS" basis, and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 😭

Acknowledgement 🧚

funding for the RFC Editor function is currently provided by the Internet Society. 💖

📦 where this page comes from