RFC 889
Internet Delay Experiments
RFC 889: Internet Delay Experiments 🎀
okay bestie, the title block first 💅 This is RFC 889, from the Network Working Group, written by D.L. Mills and dated December 1983. The header doesn't list anything it obsoletes or updates, so it's a standalone drop 🦋
This memo is giving status report on some measurement experiments, and it also has advice for TCP implementers about tweaking the retransmission timeout calculation 🔥
1. Introduction ✨
Okay so this memo describes two series of experiments that explore how things actually move around the Internet system 🌐 One series was about figuring out network delays based on packet length. The other one checked how well the TCP retransmission-timeout algorithm (the one from the standards docs) actually works 💖 Both sets ran during October to November 1983 and used loads of hosts spread all over the Internet system.
The goals were kind of two-part 💅 First, collect real data from actual network paths that could be used as a benchmark for how the Internet system performs. Second, use that data to fine-tune individual TCP implementations so they run better 🫶
The experiments ran on a specially instrumented measurement host called a Fuzzball 🧚 It's an LSI-11 running IP/TCP plus a bunch of application-layer protocols, including TELNET, FTP and SMTP mail. The collection of measurement tools includes the original PING (Packet InterNet Groper) program, which had been used for six years for all sorts of tests on the Internet system and its client nets 💀 PING can send different kinds of probe packets, including ICMP Echo messages. It processes the replies, records elapsed times and other info in a data file, and can also make real-time snapshot histograms and traces 📊
After each experiment run, the collected data got processed by another set of programs and plotted on a Peritek bit-map display with a color monitor 🌈 Those plots were super valuable for spotting and understanding the causes of network glitches and other "zoo" phenomena 🙈 Finally, summary data got pulled out and put into this memo. The raw data files, including the bit-map image files of the various plots, are available to other experimenters on request 🍓
The Fuzzballs and their local-net architecture, called DCN, have about two dozen clones scattered worldwide 🌸 There's one called DCN1 at the Linkabit Corporation offices in McLean, Virginia, and another at the Norwegian Telecommunications Administration (NTA) near Oslo, Norway. DCN1 connects to the ARPANET at the Mitre IMP through 1822 Error Control Units running over a 56-Kbps line. The NTA Fuzzball connects to the NTARE Gateway through an 1822 interface, then goes through VDH/HAP over a 9.6-Kbps line to SATNET at the Tanum (Sweden) SIMP. For most experiments in this memo you can ignore these local connection details, since only relatively small delays are involved 🙈
The remote test hosts were picked to represent canonical paths in the Internet system, and they're spread all over the world 🌍 They include hosts on the ARPANET, MILNET, MINET, SATNET, TELENET and loads of local nets reachable through these long-haul nets. To show how rich the Internet system connectivity and the experimental data base are, the memo includes data for three different paths from the ARPANET-based measurement host to London hosts: two go over different satellite links and one goes over an undersea cable 🦋
2. Packet Length Versus Delay 💜
This set of experiments was designed to find out whether delays across the Internet are significantly influenced by packet length 👀 When the intrinsic propagation delays are high compared to the time it takes to transmit one packet, you'd expect delays to not be strongly affected by packet length. That's true for satellite nets, including SATNET and WIDEBAND, and also for terrestrial nets where traffic aggregation is high, so the measured traffic is only a small slice of the total traffic on the path 🛰️ But when the intrinsic propagation delays are low and the measured traffic is the bulk of the path's traffic, you'd expect the opposite 🔥
The goal was to figure out how much TCP performance could improve by refining the retransmission-timeout algorithm to depend on packet length 💅 Another goal was to see what the delay-versus-packet-length relationship looks like on tandem paths spanning networks with super different architectures, including local-nets, terrestrial long-haul nets and satellite nets 🌈
2.1. Experiment Design 🧪
There were two sets of experiments measuring delay as a function of packet length. One was based at DCN1 and the other at NTA 💖 All experiments used ICMP Echo/Reply messages with embedded timestamps 🕰️ Each cycle sent an ICMP Echo message of a specified length, waited for the matching ICMP Reply to come back, and recorded the elapsed time (normalized to one-way delay). One experiment run, which makes one line of the table below, was made up of 512 of these volleys ✨
The length of each ICMP message came from a random-number generator, uniformly distributed between zero and 256 🎲 Lengths less than 40 were rounded up to 40. That's the minimum datagram size for an ICMP message containing timestamps, and it also just happens to be the minimum TCP segment size 😭 The maximum length was picked to avoid complications from fragmentation and reassembly, since gateways don't ordinarily fragment or reassemble ICMP messages 🧩
The collected data got plotted first as a scatter diagram on a color bit-map display. For every path involving the ARPANET, this immediately showed two distinct characteristics: one for short, single-packet messages under 126 octets, and one for long, multi-packet messages longer than that 👑 Linear regression lines were then fitted to each characteristic, and the results are in the table below. (Only one characteristic was assumed for ARPANET-exclusive paths.) The table shows, for each host, the delays in milliseconds for each type of message, along with a rate computed from those delays. The "Host ID" column names the host at the remote end of the path, and a letter suffix is used when needed to tell apart a particular run 🦋
Host Single-packet Rate Multi-packet Rate Comments
ID 40 125 (bps) 125 256 (bps)
---------------------------------------------------------------------------
DCN1 to nearby local-net hosts (calibration)
DCN5 9 13 366422 DMA 1822
DCN8 14 20 268017 Ethernet
IMP17 22 60 45228 56K 1822/ECU
FORD1 93 274 9540 9600 DDCMP base
UMD1 102 473 4663 4800 synch
DCN6 188 550 4782 4800 DDCMP
FACC 243 770 3282 9600/4800 DDCMP
FOE 608 1917 1320 9600/14.4K stat mux
DCN1 to ARPANET hosts and local nets
MILARP 61 105 15358 133 171 27769 MILNET gateway
ISID-L 166 263 6989 403 472 15029 low-traffic period
SCORE 184 318 5088 541 608 15745 low-traffic period
RVAX 231 398 4061 651 740 11781 Purdue local net
AJAX 322 578 2664 944 1081 7681 MIT local net
ISID-H 333 520 3643 715 889 6029 high-traffic period
BERK 336 967 1078 1188 1403 4879 UC Berkeley
WASH 498 776 2441 1256 1348 11379 U Washington
DCN1 to MILNET/MINET hosts and local nets
ISIA-L 460 563 6633 1049 1140 11489 low-traffic period
ISIA-H 564 841 2447 1275 1635 2910 high-traffic period
BRL 560 973 1645 1605 1825 4768 BRL local net
LON 585 835 2724 1775 1998 4696 MINET host (London)
HAWAII 679 980 2257 1817 1931 9238 a long way off
OFFICE3 762 1249 1396 2283 2414 8004 heavily loaded host
KOREA 897 1294 1712 2717 2770 19652 a long, long way off
DCN1 to TELENET hosts via ARPANET
RICE 1456 2358 754 3086 3543 2297 via VAN gateway
DCN1 to SATNET hosts and local nets via ARPANET
UCL 1089 1240 4514 1426 1548 8558 UCL zoo
NTA-L 1132 1417 2382 1524 1838 3339 low-traffic period
NTA-H 1247 1504 2640 1681 1811 8078 high-traffic period
NTA to SATNET hosts
TANUM 107 368 6625 9600 bps Tanum line
ETAM 964 1274 5576 Etam channel echo
GOONY 972 1256 6082 Goonhilly channel echo2.2 Analysis of Results 📈
The data clearly show a strong correlation between delay and length, with the longest packets showing delays two to three times those of the shortest 😭 On paths via ARPANET clones, the delay characteristic correlates more strongly with length for single-packet messages than for multi-packet messages. That fits a design that favors low delays for short messages and high throughput for longer ones 💅
Most of the runs happened during off-peak hours 🌙 In the few cases where a particular host got runs during both on-peak and off-peak hours, the comparison shows a greater dependency on packet length than on traffic shift 🙈
TCP implementors should be advised that some dependency on packet length may have to be built into the retransmission-timeout estimation algorithm to ensure good performance over lossy nets like SATNET 🫶 They should also know that some Internet paths may need stupendous timeout intervals, ranging to many seconds for the net alone, not counting the extra delays on host-system queues 💀
I want to call attention to the fact that the delays (at least for the larger packets) from ARPANET hosts (for example DCN1) to MILNET hosts (for example ISIA) are in the same ballpark as the delays to SATNET hosts (for example UCL)! 😱 I've also observed that the packet-loss rates on the MILNET path are, for now, not negligible (18 in 512 for ISIA-2). Presumably the loss is in the gateways. However, there may well be a host or two out there swamping the gateways with retransmitted data and having a funny idea of the "normal" timeout interval 👀 The recent discovery of a bug in the TOPS-20 TCP implementation, where spurious ACKs were generated at an alarming rate, seems to confirm that suspicion 🚨
3. Retransmission-Timeout Algorithm ⏱️
One of the basic features of TCP that lets it work on paths spanning many nets with widely varying delay and packet-loss characteristics is the retransmission-timeout algorithm 💖 It's sometimes known as the "RSRE Algorithm" after its original designers. The algorithm works by recording the time and initial sequence number when a segment is transmitted, then computing the elapsed time until that sequence number gets acknowledged 🧾 There are various levels of sophistication in how it can be implemented, from allowing only one such computation in progress at a time, to allowing one for each segment outstanding on the connection at a time 🔥
The retransmission-timeout algorithm is basically an estimation process 🧠 It keeps an estimate of the current roundtrip delay time and updates that estimate as new delay samples get computed. The algorithm smooths these samples, and then
The timer is the whole plot 🎀: the algorithm sets up a timeout, and if that timeout gets exceeded, the packet gets retransmitted. Choosing the parameters for this algorithm is vitally important, bestie, because they decide whether you get effective data transmission or whether the Internet gets flooded with excessive retransmissions 💀. I've been suspicious of the parameters in the specification, and in some implementations, for a long time, especially on long-delay paths that run over lossy nets 🙈. So I designed an experiment to simulate the algorithm using data collected from real paths full of pretty leaky Internet plumbing 🚰.
3.1. Experiment Design ✨
The experiment database has well over a hundred runs 📊. They used ICMP Echo/Reply messages bounced off hosts scattered all over the world 🌍. Most runs, including every single one summarized here, were 512 echo/reply cycles that lasted anywhere from several seconds to twenty minutes or so ⏱️. Other runs were built to catch network glitches and ran for several hours 🕰️.
Some runs used packets of a constant length, and others used lengths spread out from 40 to 256 octets 🎲. The maximum length was picked to avoid complications from fragmentation or reassembly by the gateways 🧩.
The goal was to simulate the packet delay distribution that TCP sees over the paths we measured 🎯. Only network delay matters here, not the queueing delays inside the hosts themselves, which can be considerable 😤. Also, only a single packet was allowed in flight at a time, so the stress on the network itself stayed minimal 🫶. Some tests ran during busy periods of network activity, and others ran during quiet hours 🌙.
Each run gave us 512 data points, and a program plotted each one as (x,y) on a colour bit-map display 🌈. The x value is the time since the experiment started, and the y value is the measured delay, normalized to the one-way delay. Then we ran the simulated retransmission-timeout algorithm on the same data and plotted its computed timeout the same way 📈. The display instantly shows how the algorithm behaves with varying traffic loads, network glitches, lost packets, and superfluous retransmissions 👀.
Each run also produced summary statistics, which are in the table below 🍓. Every line has a Host ID that identifies the run. The suffix -1 means 40-octet packets, -2 means 256-octet packets, and no suffix means lengths uniformly distributed between 40 and 256 🎀. The Lost Packets columns count the times no ICMP Reply came back within thirty seconds after the ICMP Echo was sent, which probably means one or both messages got lost 💔. The RTX Packets columns count the times the computed timeout was less than the measured delay, which would cause a superfluous retransmission 🔁. For each of those two types, the count column gives the number of instances, and the Time column gives the total accumulated time spent on the recovery action ⏳.
For reference, the Mean column is the computed mean delay of the echo/reply cycles, leaving out the cycles that involved packet loss 📐. The CoV column is the coefficient of variation 📉. Last but not least, the Eff column is the efficiency, which is the ratio of the total time accumulated while sending good data to that same time plus the lost-packet and rtx-packet time 💯.
Complete sets of runs were made for each host in the table below, for each of several selections of algorithm parameters 🗂️. The table itself reflects values, selected as described later, that were believed to be a good compromise for use on existing paths in the Internet system 💖.
Host Total Lost Packets RTX Packets Mean CoV Eff
ID Time Time Time
-------------------------------------------------------------------
DCN1 to nearby local-net hosts (calibration)
DCN5 5 0 0 0 0 11 .15 1
DCN8 8 0 0 0 0 16 .13 1
IMP17 19 0 0 0 0 38 .33 1
FORD1 86 0 0 1 .2 167 .33 .99
UMD1 135 0 0 2 .5 263 .45 .99
DCN6 177 0 0 0 0 347 .34 1
FACC 368 196 222.1 6 9.2 267 1.1 .37
FOE 670 3 7.5 21 73.3 1150 .69 .87
FOE-1 374 0 0 26 61.9 610 .75 .83
FOE-2 1016 3 16.7 10 47.2 1859 .41 .93
DCN1 to ARPANET hosts and local nets
MILARP 59 0 0 2 .5 115 .39 .99
ISID 163 0 0 1 1.8 316 .47 .98
ISID-1 84 0 0 2 1 163 .18 .98
ISID-2 281 0 0 3 17 516 .91 .93
ISID * 329 0 0 5 12.9 619 .81 .96
SCORE 208 0 0 1 .8 405 .46 .99
RVAX 256 1 1.3 0 0 499 .42 .99
AJAX 365 0 0 0 0 713 .44 1
WASH 494 0 0 2 2.8 960 .39 .99
WASH-1 271 0 0 5 8 514 .34 .97
WASH-2 749 1 9.8 2 17.5 1411 .4 .96
BERK 528 20 50.1 4 35 865 1.13 .83
DCN1 to MILNET/MINET hosts and local nets
ISIA 436 4 7.4 2 15.7 807 .68 .94
ISIA-1 197 0 0 0 0 385 .27 1
ISIA-2 615 0 0 2 15 1172 .36 .97
ISIA * 595 18 54.1 6 33.3 992 .77 .85
BRL 644 1 3 1 1.9 1249 .43 .99
BRL-1 318 0 0 4 13.6 596 .68 .95
BRL-2 962 2 8.4 0 0 1864 .12 .99
LON 677 0 0 3 11.7 1300 .51 .98
LON-1 302 0 0 0 0 589 .06 1
LON-2 1047 0 0 0 0 2044 .03 1
HAWAII 709 4 12.9 3 18.5 1325 .55 .95
OFFICE3 856 3 12.9 3 10.3 1627 .54 .97
OFF3-1 432 2 4.2 2 6.9 823 .31 .97
OFF3-2 1277 7 39 3 41.5 2336 .44 .93
KOREA 1048 3 14.5 2 18.7 1982 .48 .96
KOREA-1 506 4 8.6 1 2.2 967 .18 .97
KOREA-2 1493 6 35.5 2 19.3 2810 .19 .96
DCN1 to TELENET hosts via ARPANET
RICE 677 2 6.8 3 12.1 1286 .41 .97
RICE-1 368 1 .1 3 2.3 715 .11 .99
RICE-2 1002 1 4.4 1 9.5 1930 .19 .98
DCN1 to SATNET hosts and local nets via ARPANET
UCL 689 9 26.8 0 0 1294 .21 .96
UCL-1 623 39 92.8 2 5.3 1025 .32 .84
UCL-2 818 4 13.5 0 0 1571 .15 .98
NTA 779 12 38.7 1 3.7 1438 .24 .94
NTA-1 616 24 56.6 2 5.3 1083 .25 .89
NTA-2 971 19 71.1 0 0 1757 .2 .92
NTA to SATNET hosts and local nets
TANUM 110 3 1.6 0 0 213 .41 .98
GOONY 587 19 44.2 1 2.9 1056 .23 .91
ETAM 608 32 76.3 1 3.1 1032 .29 .86
UCL 612 5 12.6 2 8.5 1154 .24 .96Note: * means randomly distributed packets during periods of high ARPANET activity. The same entry without the * means randomly distributed packets during periods of low ARPANET activity 🌐.
3.2 Discussion of Results 🔍
Just from looking at the bit-map display, it's super obvious that the delay distribution is more-or-less Poisson distributed around a fairly narrow range 📊, with some important exceptions 😭. The exceptions are these occasional spasms where one or more packets get delayed many times longer than the typical value 💥. People had noticed glitches like this before on paths that go through ARPANET and SATNET 🛰️, but the real impact they have on the timeout algorithm is way bigger than I expected 😳. Here's what usually happens: the algorithm hits a short burst of long-delay packets after a pretty long stretch of well-mannered behaviour, and it takes WAY too long to adapt to the spasm 🐢. That invites a bunch of superfluous retransmissions, which leads to congestion 🚧.
How many long-delay bursts (glitches) showed up changed a lot during the experiments 🎢. Some runs were glitch-free, but most had at least one glitch in 512 echo/reply volleys 🎾. Glitches didn't correlate well with increases in baseline delay caused by traffic surges 🌊, and they didn't correlate well with instances of packet loss either 🤷♀️. I didn't notice any particular periodicity, like you might expect with regular pinging, for example 🏓. I didn't process the data specially to look for it, though 🙃.
We didn't correct for packet length in any of these experiments, even though the first set of experiments described previously suggested we should 🙈. That might get done in a future set of experiments 🔮. The algorithm does cope well with constant-length packets and with randomly distributed packet lengths between 40 and 256 octets, as the table shows 💅. Future experiments may involve bursts of short packets followed by bursts of longer ones, so that the speed of adaptation of the algorithm can be directly determined 🎯.
One particularly interesting experiment involved the FOE host (FORD-FOE). It's located in London and reached via a 14.4-Kbps undersea cable and a statistical multiplexor 🌊. The multiplexor adds a moderate mean delay, but with an extremely large delay dispersion 😵. The specified retransmission-timeout algorithm had a really hard time with this circuit, which is what you'd expect 😩. With the improvements described below, though, TCP performance was acceptable ✨. It's unlikely that many instances of such ornery circuits will show up in the Internet system, but it's super comforting to know that the algorithm can deal with them effectively 🫶.
3.3. Improvments to the Algorithm 🛠️
The specified retransmission-timeout algorithm is really a first-order linear recursive filter 🧮. It's characterized by two parameters: a weighting factor F and a threshold factor G 🎛️. For each measured delay sample R, the delay estimator E gets updated like this:
E = F*E + (1 - F)*RThen, if an interval equal to G*E expires after a packet is transmitted, the packet is retransmitted ⏰. The current TCP specification suggests values in the range 0.8 to 0.9 for F and 1.5 to 2.0 for G 📜. These values have been believed reasonable up to now over ARPANET and SATNET paths 🌸.
I found that a simple change to the algorithm made a seriously worthwhile difference in efficiency 🤯. The change is to use two values of F: one (F1) when R < E in the expression above, and the other (F2) when R >= E, with F1 > F2 💜. The effect is that the algorithm becomes more responsive to upward-going trends in delay and less responsive to downward-going trends 📈📉. After a bunch of trials, I concluded that F1 = 15/16 and F2 = 3/4 (with G = 2) gave the best all-around performance 🏆. On some paths (FOE, ISID, ISIA) the results were better by roughly ten percent in efficiency, compared to the values in typical implementations where F = 7/8 and G = 2 🚀. On most paths the results were better by five percent 🎉, while on a couple of them (FACC, UCL) the results were worse by a few percent 😬.
There was no clear-cut gain from fiddling with G 🤷♀️. The value G = 2 seemed to be the best overall compromise 👑. Note that increasing G makes
superfluous retransmissions less likely, but it bumps up the total delay whenever packets get lost 😭 so it's a tradeoff, bestie, and you gotta pick your G wisely 🎀. Also, ngl, cranking F2 up too much tends to cause overshoot during network glitches, and it leads to the same result 💀. The table above was built with F1 = 15/16, F2 = 3/4 and G = 2 💅✨
If you know your signal-detection theory, you'll totally clock that my suggestion is basically an ordinary peak-detector circuit 🔌🌈. F1 is the discharge time-constant, and F2 is the charge time-constant. G is a "squelch" threshold, like the ones in voice-operated switches, for example 🎙️. And some wag may even go on to suggest that a network glitch should be called a netspurt 🫶🙈
Appendix. Index of Test Hosts 🦋
Here's the whole test-host lineup, sorted by which network they're chatting on 🧚💜
Name Address NIC Host Name
-------------------------------------
DCN1 to nearby local-net hosts (calibration)
DCN5 128.4.0.5 DCN5
DCN8 128.4.0.8 DCN8
IMP17 10.3.0.17 DCN-GATEWAY
FORD1 128.5.0.1 FORD1
UMD1 128.8.0.1 UMD1
DCN6 128.4.0.6 DCN6
FACC 128.5.32.1 FORD-WDL1
FOE 128.5.0.15 FORD-FOE
DCN1 to ARPANET hosts and local nets
MILARP 10.2.0.28 ARPA-MILNET-GW
ISID 10.0.0.27 USC-ISID
SCORE 10.3.0.11 SU-SCORE
RVAX 128.10.0.2 PURDUE-MORDRED
AJAX 18.10.0.64 MIT-AJAX
WASH 10.0.0.91 WASHINGTON
BERK 10.2.0.78 UCB-VAX
DCN1 to MILNET/MINET hosts and local nets
ISIA 26.3.0.103 USC-ISIA
BRL 192.5.21.6 BRL-VGR
LON 24.0.0.7 MINET-LON-EM
HAWAII 26.1.0.36 HAWAII-EMH
OFFICE3 26.2.0.43 OFFICE-3
KOREA 26.0.0.117 KOREA-EMH
DCN1 to TELENET hosts via ARPANET
RICE 14.0.0.12 RICE
DCN1 to SATNET hosts and local nets via ARPANET
UCL 128.16.9.0 UCL-SAM
NTA 128.39.0.2 NTARE1
NTA to SATNET hosts and local nets
TANUM 4.0.0.64 TANUM-ECHO
GOONY 4.0.0.63 GOONHILLY-ECHO
ETAM 4.0.0.62 ETAM-ECHO📦 where this page comes from
- ietf: rfc889 1983-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms