RFC 8257

Data Center TCP (DCTCP): TCP Congestion Control for Data Centers

📅 published October 2017, from ietf

🎀 RFC 8257: Data Center TCP (DCTCP): TCP Congestion Control for Data Centers ✨

okay bestie, let's get into it 💅 This is an Informational RFC (so it's not a Standards Track thing), published in October 2017, and it was written by S. Bensley, D. Thaler, P. Balasubramanian (all from Microsoft), L. Eggert (from NetApp), and G. Judd (from Morgan Stanley). It's in the ISSN: 2070-1721 series, and it doesn't obsolete or update any other RFCs 🦋

Here's the vibe: DCTCP is a TCP congestion control scheme built for data-center traffic 🌸 It upgrades ECN processing so the sender estimates the fraction of bytes that hit congestion, instead of just noticing that some congestion happened, and then scales the congestion window off that estimate. The result is high-burst tolerance, low latency, and high throughput with shallow-buffered switches 🔥 The memo also talks about deployment issues when DCTCP and conventional TCP coexist, the missing negotiation mechanism between sender and receiver, and some possible mitigations 💜

Also, the big warning: DCTCP is for controlled environments like data centers, and it must not be deployed over the public Internet without additional measures 🙈

Status of This Memo 🌈

This document is not an Internet Standards Track specification; it is published for informational purposes.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 7841.

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8257.

Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

1. Introduction ✨

okay so big data centers need tons of network switches to connect all their servers 🖥️ 🧚 Cheap switches save a ton of money, but they usually have small queues, which means they're way more likely to drop packets when things get congested 😭

Data center traffic is a mix of short flows and long flows 🌸 The short ones need low latency, and the long ones need high throughput. Data centers also get incast bursts, where a bunch of servers all send traffic to one server at the same time 💅 A super common example is MapReduce [MAPREDUCE]: the workers finish around the same time and all reply to the master node at once 🎀

All of this creates some tension for the switch queue 🫶

  • The queue has to be short enough so it doesn't add excessive latency for short flows.
  • The queue has to be long enough to hold enough data for long flows to fill the path capacity.
  • The queue has to be long enough to soak up incast bursts without excessive packet loss.

Standard TCP congestion control [RFC5681] basically waits for packet loss to figure out there's congestion, and that just doesn't cut it here 😭 First, short flows start suffering unacceptable latency before any packets get dropped. Second, by the time TCP congestion control kicks in on the senders, most of the incast burst is already gone 💀

[RFC3168] describes a way to use Explicit Congestion Notification (ECN) from the switches to detect congestion 🦋 The catch is that it only tells you congestion exists, not how bad it is. With mild congestion, the TCP congestion window gets cut way too hard, and that unnecessarily slows down the long flows 🙈

That's where DCTCP comes in ✨ It changes traditional ECN processing by estimating the fraction of bytes that encounter congestion, rather than just detecting that some congestion happened. Then it scales the TCP congestion window based on that estimate 🔥 This gives high-burst tolerance, low latency, and high throughput with shallow-buffered switches 💜 DCTCP is a tweak to how a conventional TCP processes ECN, and it still requires standard TCP congestion control to handle packet loss 🌈

DCTCP should only be deployed inside a single intra-data-center environment, where the endpoints and the switching fabric are all under one administrative domain. DCTCP MUST NOT be deployed over the public Internet without additional measures, as detailed in Section 5 🫶

The goal of this Informational RFC is to document DCTCP as a new approach (widely implemented and deployed already 🍓) for TCP congestion control in data centers. The IETF TCPM Working Group reached consensus that a DCTCP standard would need more work. Precisely documenting running code lets follow-up Experimental or Standards Track RFCs come through the IETF stream 👑

This document describes DCTCP as implemented in Microsoft Windows Server 2012 [WINDOWS]. The Linux [LINUX] and FreeBSD [FREEBSD] operating systems have also implemented DCTCP support in a way that is believed to follow this document. Deployment experiences with DCTCP have been documented in [MORGANSTANLEY] 🧚

2. Terminology 💜

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Normative language describes how necessary the various aspects of a DCTCP implementation are for interoperability 🎀 But even compliant implementations without the measures in Sections 4-6 would still only be safe to deploy in controlled environments, i.e., not over the public Internet 🙈

3. DCTCP Algorithm 🌸

There are three parts to the DCTCP algorithm 💅

  • The switches (or other intermediate devices in the network) detect congestion and set the Congestion Encountered (CE) codepoint in the IP header.
  • The receiver echoes the congestion information back to the sender, using the ECN-Echo (ECE) flag in the TCP header.
  • The sender computes a congestion estimate and reacts by reducing the TCP congestion window (cwnd) accordingly.

3.1. Marking Congestion on the L3 Switches and Routers ⭐

The Layer 3 (L3) switches and routers in a data-center fabric tell end nodes about congestion by setting the CE codepoint in the IP header, as specified in Section 5 of [RFC3168] 🦋 For example, the switches may be configured with a congestion threshold. When a packet arrives at a switch and its queue length is greater than that threshold, the switch sets the CE codepoint in the packet 🫶 For example, Section 3.4 of [DCTCP10] suggests threshold marking with a threshold of K > (RTT * C)/7, where C is the link rate in packets per second 🔥 In typical deployments, the marking threshold is set to a small value to keep the average queueing delay short 🌈

However, the actual marking algorithm is an implementation detail of the switch and generally won't be known to the sender and receiver 🙈 So the sender and receiver should not assume that a particular marking algorithm is implemented by the switching fabric 💀

3.2. Echoing Congestion Information on the Receiver 🎀

okay so, according to Section 6.1.3 of [RFC3168], the receiver sets the ECE flag if any of the packets being acknowledged had the CE codepoint set. It then keeps setting ECE until it gets a packet with the Congestion Window Reduced (CWR) flag set. 💅 But DCTCP needs way more detailed info. The sender needs to be able to figure out how many bytes it sent that actually hit congestion, so the scheme in [RFC3168] just doesn't cut it 😭

One possible fix is to ACK every single packet and set ECE in the ACK if and only if the CE codepoint was set on the packet being acknowledged. The problem is that this kills delayed ACKs, which are a huge performance win in data centers 💀 If the delayed ACK frequency is n, then an ACK gets generated every n packets.

The typical value of n is 2 🌸, but ACK throttling or packet-coalescing tricks meant to boost performance could change it.

Instead, DCTCP introduces a new Boolean TCP state variable called DCTCP Congestion Encountered (DCTCP.CE). It starts out false and lives in the Transmission Control Block (TCB). When sending an ACK, the ECE flag MUST be set if and only if DCTCP.CE is true. When receiving packets, the CE codepoint MUST be processed as follows:

  1. If the CE codepoint is set and DCTCP.CE is false, set DCTCP.CE to true and send an immediate ACK.
  2. If the CE codepoint is not set and DCTCP.CE is true, set DCTCP.CE to false and send an immediate ACK.
  3. Otherwise, ignore the CE codepoint.

Since the immediate ACK reflects the new DCTCP.CE state, it may acknowledge any previously unacknowledged packets in the old state. That can lead to an incorrect rate computation at the sender, per Section 3.3. To avoid this, an implementation MAY choose to send two ACKs: one for previously unacknowledged packets and another acknowledging the most recently received packet. ✨

Receiver handling of the CWR bit is also per [RFC3168] (including [Err3639]). That means that when a segment shows up with both the CE and CWR bits set, CWR gets processed first and then CE gets processed. 🦋

                         Send immediate
                         ACK with ECE=0
             .-----.     .--------------.     .-----.
Send 1 ACK  /      v     v              |     |      \
 for every |     .------------.    .------------.     | Send 1 ACK
 n packets |     | DCTCP.CE=0 |    | DCTCP.CE=1 |     | for every
with ECE=0 |     '------------'    '------------'     | n packets
            \      |     |              ^     ^      /  with ECE=1
             '-----'     '--------------'     '-----'
                          Send immediate
                          ACK with ECE=1

Figure 1: ACK Generation State Machine

3.3. Processing Echoed Congestion Indications on the Sender 🦋

The sender estimates the fraction of bytes sent that encountered congestion. The current estimate lives in a new TCP state variable, DCTCP.Alpha. It is initialized to 1 and SHOULD be updated as follows:

DCTCP.Alpha = DCTCP.Alpha * (1 - g) + g * M

where:

  • g is the estimation gain, a real number between 0 and 1. Picking g is left up to the implementation. See Section 4 for further considerations. 🌈
  • M is the fraction of bytes sent that encountered congestion during the previous observation window. The observation window is chosen to be approximately the Round-Trip Time (RTT). In particular, an observation window ends when all bytes in flight at the beginning of the window have been acknowledged.

To update DCTCP.Alpha, the TCP state variables defined in [RFC0793] get used, and three additional TCP state variables get introduced:

  • DCTCP.WindowEnd: the TCP sequence number threshold when one observation window ends and another is to begin; initialized to SND.UNA.
  • DCTCP.BytesAcked: the number of sent bytes acknowledged during the current observation window; initialized to 0.
  • DCTCP.BytesMarked: the number of bytes sent during the current observation window that encountered congestion; initialized to 0.

The congestion estimator on the sender MUST process acceptable ACKs as follows:

  1. Compute the bytes acknowledged (TCP Selective Acknowledgment (SACK) options [RFC2018] are ignored for this computation): BytesAcked = SEG.ACK - SND.UNA
  2. Update the bytes sent: DCTCP.BytesAcked += BytesAcked
  3. If the ECE flag is set, update the bytes marked: DCTCP.BytesMarked += BytesAcked
  4. If the acknowledgment number is less than or equal to DCTCP.WindowEnd, stop processing. Otherwise, the end of the observation window has been reached, so proceed to update the congestion estimate as follows:
  5. Compute the congestion level for the current observation window: M = DCTCP.BytesMarked / DCTCP.BytesAcked
  6. Update the congestion estimate: DCTCP.Alpha = DCTCP.Alpha * (1 - g) + g * M
  7. Determine the end of the next observation window: DCTCP.WindowEnd = SND.NXT
  8. Reset the byte counters: DCTCP.BytesAcked = DCTCP.BytesMarked = 0
  9. Rather than always halving the congestion window as described in [RFC3168], the sender SHOULD update cwnd as follows: cwnd = cwnd * (1 - DCTCP.Alpha / 2)

Just like [RFC3168] says, DCTCP doesn't react to congestion indications more than once for every window of data. The setting of the CWR bit is also as per [RFC3168]. This is required for interoperation with classic ECN receivers due to potential misconfigurations. 🫶

3.4. Handling of Congestion Window Growth 🌸

A DCTCP sender grows its congestion window the same way conventional TCP does. Slow start and congestion avoidance algorithms are handled as specified in [RFC5681].

3.5. Handling of Packet Loss 💀

A DCTCP sender MUST react to loss episodes the same way conventional TCP does, including the fast retransmit and fast recovery algorithms, as specified in [RFC5681]. For cases where the packet loss is inferred and not explicitly signaled by ECN, cwnd and other state variables like ssthresh MUST be changed the same way a conventional TCP would have changed them. As with ECN, a DCTCP sender will only reduce cwnd once per window of data across all loss signals. Just as specified in [RFC5681], upon a timeout, cwnd MUST be set to no more than the loss window (1 full-sized segment), regardless of previous cwnd reductions in a given window of data.

Loss handling is where the real drama is, bestie. The sender still has to follow conventional TCP here, so don't get creative with the reductions 🙈

3.6. Handling of SYN, SYN-ACK, and RST Packets 🧚

If SYN, SYN-ACK, and RST packets for DCTCP connections have the ECN-Capable Transport (ECT) codepoint set in the IP header, they get the same treatment as other DCTCP packets when a switching fabric forwards them under load. Lack of ECT in these packets can result in a higher drop rate, depending on how the switching fabric is configured. Hence, for DCTCP connections, the sender SHOULD set ECT for SYN, SYN-ACK, and RST packets. A DCTCP receiver ignores CE codepoints set on any SYN, SYN-ACK, or RST packets. 💜

4. Implementation Issues 🛠️

4.1. Configuration of DCTCP 🎀

An implementation needs to know when to use DCTCP. Data-center servers may need to talk to endpoints outside the data center, where DCTCP is unsuitable or unsupported. So a global configuration setting to enable DCTCP generally won't cut it. DCTCP provides no mechanism for negotiating its use. That means additional management and configuration functionality is needed to make sure DCTCP isn't used with non-DCTCP endpoints. 🙅‍♀️

Known solutions rely on either configuration or heuristics. Heuristics need to let endpoints individually enable DCTCP, so a DCTCP sender is always paired with a DCTCP receiver. One approach is to enable DCTCP based on the IP address of the remote endpoint. Another approach is to detect connections that transmit within the bounds of a data center. For example, an implementation could support automatic selection of DCTCP if the estimated RTT is less than a threshold (like 10 msec) and ECN is successfully negotiated. The idea is that if the RTT is low, the two endpoints are likely in the same data-center network. 🔥

[RFC3168] forbids the ECN-marking of pure ACK packets because TCP can't mitigate ACK-path congestion. RFC 3168 also forbids ECN-marking of retransmissions, window probes, and RSTs. However, dropping all these control packets (instead of ECN-marking them) has considerable performance disadvantages. It is RECOMMENDED that an implementation provide a configuration knob that causes ECT to be set on such control packets, which can be used in environments where those concerns don't apply. See [ECN-EXPERIMENTATION] for details. 👑

It's useful to implement DCTCP as an additional action on top of an existing congestion control algorithm like Reno [RFC5681]. The DCTCP implementation MAY also allow configuration of resetting the value of DCTCP.Alpha as part of processing any loss episodes.

4.2. Computation of DCTCP.Alpha 🧮

As noted in Section 3.3, the implementation will need to choose a suitable estimation gain. [DCTCP10] provides a theoretical basis for selecting the gain. However, it may be more practical to use experimentation to select a suitable gain for a particular network and workload. A fixed estimation gain of 1/16 is used in some implementations. (It should be noted that values of 0 or 1 for g result in problematic behavior: g=0 fixes DCTCP.Alpha to its initial value, and g=1 sets it to M without any smoothing. 💀)

The DCTCP.Alpha computation, per the formula in Section 3.3, involves fractions. An efficient kernel implementation MAY scale the DCTCP.Alpha value for efficient computation using shift operations. For example, if the implementation chooses g as 1/16, multiplications of DCTCP.Alpha by g become right-shifts by 4. A scaling implementation SHOULD ensure that DCTCP.Alpha is able to reach 0 once it falls below the smallest shifted value (16 in the above example). At the other extreme, a scaled update needs to ensure DCTCP.Alpha does not exceed the scaling factor, which would be equivalent to greater than 100% congestion. So, DCTCP.Alpha MUST be clamped after an update. ✨

This results in the following computations replacing steps 5 and 6 in Section 3.3, where SCF is the chosen scaling factor (65536 in the example), and SHF is the shift factor (4 in the example):

  1. Compute the congestion level for the current observation window:
ScaledM = SCF * DCTCP.BytesMarked / DCTCP.BytesAcked
  1. Update the congestion estimate:
if (DCTCP.Alpha >> SHF) == 0, then DCTCP.Alpha = 0

DCTCP.Alpha += (ScaledM >> SHF) - (DCTCP.Alpha >> SHF)

if DCTCP.Alpha > SCF, then DCTCP.Alpha = SCF

5. Deployment Issues 🚀

Okay so bestie, DCTCP and conventional TCP congestion control do NOT vibe together in the same network 😭 In typical DCTCP deployments, the marking threshold in the switching fabric is set to a very low value to reduce queueing delay, and a relatively small amount of congestion will exceed the marking threshold. During those moments, conventional TCP takes a massive L 📉 and drops packet loss and quickly and drastically reduces cwnd. DCTCP, on the other hand, is giving gradual: it uses the fraction of marked packets to reduce cwnd more smoothly 🦋 So the rate reduction in DCTCP is much slower than conventional TCP, and DCTCP traffic ends up gaining a larger share of the capacity compared to conventional TCP traffic on the same path. Lowkey unfair, ngl 💅 If the traffic in the data center is a mix of conventional TCP and DCTCP, it is RECOMMENDED that DCTCP traffic be segregated from conventional TCP traffic 🎀 [MORGANSTANLEY] describes a deployment that uses the IP Differentiated Services Codepoint (DSCP) bits to segregate the network, so Active Queue Management (AQM) [RFC7567] is applied to DCTCP traffic while TCP traffic is managed via drop-tail queueing 🧚

Deployments should also think about segregation of non-TCP traffic 🌸 Today's commodity switches let you configure different marking/drop profiles for non-TCP and non-IP packets. Non-TCP and non-IP packets SHOULD be able to pass through such switches, unless they really run out of buffer space 💖

Since DCTCP relies on congestion marking by the switches, DCTCP's potential can only be realized in data centers where the entire network infrastructure supports ECN ✨ The switches may also support configuration of the congestion threshold used for marking. The proposed parameterization can be configured with switches that implement Random Early Detection (RED) [RFC2309]. [DCTCP10] provides a theoretical basis for selecting the congestion threshold, but, just like with the estimation gain, it may be more practical to rely on experimentation or simply use the default configuration of the device 🙈 DCTCP will revert to loss-based congestion control when packet loss is experienced (for example, when transiting a congested drop-tail link, or a link with an AQM drop behavior) 🍓

DCTCP requires changes on both the sender and the receiver, so both endpoints MUST support DCTCP 🫶 Furthermore, DCTCP provides no mechanism for negotiating its use, so both endpoints MUST be configured through some out-of-band mechanism to use DCTCP 👑 A variant of DCTCP that can be deployed unilaterally and that only requires standard ECN behavior has been described in [ODCTCP] and [BSDCAN], but it requires additional experimental evaluation 🔥

6. Known Issues 💀

DCTCP relies on the sender's ability to reconstruct the stream of CE codepoints received by the remote endpoint. To make that work, DCTCP avoids using a single ACK packet to acknowledge segments received both with and without the CE codepoint set. However, if one or more ACK packets are dropped, it's possible that a subsequent ACK will cumulatively acknowledge a mix of CE and non-CE segments 😭 This will, of course, result in a less-accurate congestion estimate. There are some potential considerations:

  • Even with an inaccurate congestion estimate, DCTCP may still perform better than [RFC3168] ✨
  • If the estimation gain is small relative to the packet loss rate, the estimate may not be too inaccurate 🦋
  • If ACK packet loss mostly occurs under heavy congestion, most drops will happen during an unbroken string of CE packets, and the estimate will be unaffected 🌈

However, the effect of packet drops on DCTCP under real-world conditions has not been analyzed 🙈

DCTCP provides no mechanism for negotiating its use. The effect of using DCTCP with a standard ECN endpoint has been analyzed in [ODCTCP] and [BSDCAN]. Furthermore, it is possible that other implementations may also modify behavior in the [RFC3168] style without negotiation, causing further interoperability issues 💅

Much like standard TCP, DCTCP is biased against flows with longer RTTs. A method for improving the RTT fairness of DCTCP has been proposed in [ADCTCP], but it requires additional experimental evaluation 🧚

7. Security Considerations 🔐

Okay, security time, and this part is SERIOUS bestie 😤 DCTCP enhances ECN, so it inherits the general security considerations discussed in [RFC3168], although additional mitigation options exist due to the limited intra-data-center deployment of DCTCP 🛡️

The processing changes introduced by DCTCP do not exacerbate the considerations in [RFC3168] or introduce new ones. In particular, with either algorithm, the network infrastructure or the remote endpoint can falsely report congestion and, thus, cause the sender to reduce cwnd. However, this is no worse than what can be achieved by simply dropping packets 💀

[RFC3168] requires that a compliant TCP MUST NOT set ECT on SYN or SYN-ACK packets. [RFC5562] proposes setting ECT on SYN-ACK packets but maintains the restriction of no ECT on SYN packets. Both of these RFCs prohibit ECT in SYN packets due to security concerns regarding malicious SYN packets with ECT set. However, these RFCs are intended for general Internet use; they do not directly apply to a controlled data-center environment 🏢 The security concerns addressed by both of these RFCs might not apply in controlled environments like data centers, and it might not be necessary to account for the presence of non-ECN servers ✨ Beyond the security considerations related to virtual servers, additional security can be imposed in the physical servers to intercept and drop traffic resembling an attack 🛡️💜

8. IANA Considerations 📋

This document does not require any IANA actions 🌸

9. References 📚

9.1. Normative References

9.2. Informative References

Acknowledgments ✨

The DCTCP algorithm was first proposed and analyzed in [DCTCP10] 💖 by Mohammad Alizadeh, Albert Greenberg, Dave Maltz, Jitu Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. Iconic team, no notes 👑

We'd also love to thank Andrew Shewmaker 🫶 for spotting the problem with clamping DCTCP.Alpha and coming up with a fix for it. Main character energy for real 💅

Lars Eggert received funding from the European Union's Horizon 2020 research and innovation program 2014-2018 under grant agreement No. 644866 ("SSICLOPS"). 🌸 Quick but important bit: this document only reflects the authors' views, and the European Commission is not responsible for any use that may be made of the information it contains. Okay, that's the boring legal part, I promise we're done 🙈

Authors' Addresses 🦋

Here's where you can find the authors. Stephen Bensley is at Microsoft, and Dave Thaler and Praveen Balasubramanian are at Microsoft too, so it's a whole Microsoft squad 💜

Stephen Bensley
Microsoft
One Microsoft Way
Redmond, WA  98052
United States of America

Phone: +1 425 703 5570
Email: sbens@microsoft.com
Dave Thaler
Microsoft

Phone: +1 425 703 8835
Email: dthaler@microsoft.com
Praveen Balasubramanian
Microsoft

Phone: +1 425 538 2782
Email: pravb@microsoft.com
Lars Eggert
NetApp
Sonnenallee 1
Kirchheim  85551
Germany

Phone: +49 151 120 55791
Email: lars@netapp.com
URI:   http://eggert.org/
Glenn Judd
Morgan Stanley

Phone: +1 973 979 6481
Email: glenn.judd@morganstanley.com

That's the whole RFC, bestie! 🌈✨ Thanks for reading all the way to the end 🎀

📦 where this page comes from