RFC 957
Experiments in network clock synchronization
RFC 957 🎀 Experiments in Network Clock Synchronization
- Group: Network Working Group 💖
- Number: RFC 957
- Author: D.L. Mills, M/A-COM Linkabit 👑
- Date: September 1985 🌸
- Category: The header doesn't label a category. The Status section says it's here to talk about experiments and ask for discussion and suggestions for improvements ✨
- Obsoletes / Updates: The header doesn't list any RFCs this one obsoletes or updates 🦋
Status of this Memo
okay so this one talks about some experiments in clock synchronization in the ARPA-Internet community, and it's basically asking everyone for discussion and suggestions for improvements 💅 distribution of this memo is unlimited, so share it with your whole group chat 🫶
Table of Contents 🌈
1. Introduction
2. Design of the Synchronization Algorithm
2.1. The Logical Clock
2.2. Linear Phase Adjustments
2.3. Nonlinear Phase Adjustments
3. Synchronizing Network Clocks
3.1. Reference Clocks and Reference Hosts
3.2. Distribution of Timing Information
4. Experimental Validation of the Design
4.1. Experiment Design
4.2. Experiment Execution
4.3. Discussion of Results
4.3.1. On Power-Grid Clocks
4.3.2. On Clocks Synchronized via Network Links
4.3.3. On the Accuracy of Radio Clocks
4.3.3.1. The Spectracom 8170 WWVB Radio Clock
4.3.3.2. The True Time 468-DC GOES Radio Clock
4.3.3.3. The Heath GC-1000 WWV Radio Clock
4.3.4. On Handling Disruptions
4.4. Additional Experiments
5. Summary and Conclusions
6. References
List of Figures
Figure 1. Clock Registers
Figure 2. Network Configuration
List of Tables
Table 1. Experiment Hosts
Table 2. Link Measurements
Table 3. First Derivative of Delay
Table 4. GOES Radio Clock Offsets
Table 5. WWV Radio Clock Offsets
Table 6. ISI-MCON-GW Clock Statistics
Table 7. LL-GW Clock Statistics
Table 8. LL-GW Clock Statistics1. Introduction 🕰️
bestie, one service that people keep ignoring in computer network design is a high-quality, time-of-day clock that can make accurate timestamps with tiny leftover errors compared to the intrinsic one-way network delays 😭 a clock like that is super useful for tracing the progress of complex transactions, synchronizing cached databases, monitoring network performance, and isolating problems. it's giving main character energy ✨
the Internet protocol suite already has a few ways to record and send the time an event happens. there's the ICMP Timestamp message [6], the Time Protocol [7], the Daytime protocol [8], and the IP Timestamp option [9]. a new Network Time Protocol [12] has also been proposed 🧚 you can find more info on network time sync in the References at the end of this doc. synchronization protocols are described in [3] and [12], and synchronization algorithms are in [2], [5], [10] and [11]. experimental results on measured roundtrip delays in the Internet are discussed in [4]. and a comprehensive mathematical treatment of clock synchronization is in [1] 💜
aside: the original Introduction repeats its opening paragraph with some different citations, so here's that second version too, same vibe 🙈
the Internet protocol suite already has a few ways to record and send the time an event happens: the ICMP Timestamp message [6], the Time protocol [7], the Daytime protocol [8], and the IP Timestamp option [9]. issues on time synchronization are discussed in [4], and synchronization algorithms are in [2] and [5]. experimental results on measured roundtrip delays in the Internet are discussed in [2]. a comprehensive mathematical treatment of the subject is in [1], while an interesting discussion on mutual-synchonization techniques is in [10] 🦋
there are a few ways to make accurate timestamps. one is to give every service point an accurate, machine-readable clock that's synced to a central reference, like the National Bureau of Standards (NBS) 🍓 these clocks are easy to get in several models, with accuracies ranging from a few hundred milliseconds down to less than a millisecond, and they usually sync to special ground-based or satellite-based radio broadcasts. the clocks cost somewhere between $300 and $3000, which can often be justified, but they all need carefully sited antennas far away from computer-generated electromagnetic noise, plus shielded connections to the clocks. also, these clocks can need a long sync period when they power up, so a battery-backup power supply is required for reliable service during power interruptions 🔋
if the propagation delays in the network are stable or can be predicted accurately, a central server with a clock like that can make timestamps when remote service points ask for them 🌸 the problem is there are lots of times when the network round trip to get a timestamp would be intolerable, like timestamping a message right before it's sent. plus, propagation delays usually can't be predicted with the precision needed, because of probabilistic queuing and channel-contention delays 😩
in principle, you can give each service point a clock that's accurate enough by using a stable, crystal-controlled clock, and then correcting it from time to time with messages from a central server 💖 suitable inexpensive, crystal-controlled clock interfaces are available for pretty much any computer. the interesting problem left is designing the synchronization algorithm and the protocol used to send those corrections. this document describes one such design and assesses how well it performs 🔥 the design has been built in as an integral part of the network routing and control protocols of the Distributed Computer Network (DCnet) architecture [5], and clones of it have been set up at several sites in the US and Europe 🌍 these protocols have been in use since 1979, and they've been continuously tested and refined ever since 👑
2. Design of the Synchronization Algorithm 💅
the synchronization algorithm is distributed by nature, with protocol peers living in every host on the network 🫶 peers talk to each other in pairs using special control messages called Hello messages, which get swapped periodically over the ordinary data links between them. the Hello messages carry the info each host needs to work out the delay and offset between its own local clock and the clock of every other host on the network. they also drive the routing algorithm, so they're doing double duty ✨
the algorithm has a bunch of features that make the local clock more accurate and stable when a host or link fails 🛡️ this section just sums up the design. the full design details are in [5], along with a formal description of the Hello protocol 🧠
2.1. The Logical Clock 🌟
in the DCnet model, each service point (or host) has a hardware clock, usually as an off-the-shelf interface 🔧 using that plus some software registers, they build a logical clock. it includes a 48-bit Clock Register that ticks at a 1000 Hz rate, a 32-bit Clock-Adjust Register that's used to slew the Clock Register in response to raw corrections received over the net, and a Counter Register, which some interface designs use as an auxiliary counter. the layout and decimal point of these registers are in Figure 1 below 📐
Clock Register
0 16 32
+---------------+---------------+---------------+
| | | |
+---------------+---------------+---------------+
A
decimal point
Clock-Adjust Register
0 16
+---------------+---------------+
| | |
+---------------+---------------+
A
decimal point
Counter Register
0 16
+---------------+
| |
+---------------+
A
decimal point
Figure 1. Clock Registersthe Clock Register and Clock-Adjust Register are implemented in memory 🧠 in typical clock interface designs like the DEC KMV11-A, the Counter Register is implemented in the interface as a buffered counter driven by a crystal oscillator. a counter overflow gets signalled by an interrupt, which bumps the Clock Register at bit 15 and carries the increment through as needed 💫 to get the time of day, you read the Counter Register (which doesn't mess with its counting) and add its value to the Clock Register with the decimal points lined up 🎀
in other interface designs, like the simple LSI-11 event-line mechanism, every tick of the clock gets signalled by an interrupt at intervals of 10, 16-2/3 or 20 ms, depending on the interface and clock source ⏰ when that happens, the right number of milliseconds (expressed to 32 bits of precision) gets added to the Clock Register with the decimal points lined up 🍬
ok real talk, great care in operating system design is needed to keep the full accuracy of timestamps for the application program 😤 the program has to be protected from pre-emption, excessive device latencies, and so on. also, the execution time of any sequence that runs with the interrupt system disabled must be strictly limited 🚫 the PDP11 operating system most often used in the DCnet is the "Fuzzball" operating system, and since it was built with all of this foremost in mind, it's been especially useful for detailed network performance testing and evaluation 🏆 other systems, especially the various Unix systems, haven't been found accurate enough for this purpose 💀
left uncorrected, the host's logical clock runs at the rate of its intrinsic oscillator, whether that's derived from a crystal or the power frequency 🕐 the correction mechanism uses the Clock-Adjust Register, which gets updated from time to time as raw corrections come in. those corrections are computed from roundtrip delays and offsets derived from the routing algorithm (which we'll get to later in this document). they're relatively noisy compared to the precision of the logical clock, so a carefully designed smoothing mechanism keeps things stable and isolated from big transients caused by link retransmissions, host reboots, and similar disruptions 🌊 that's the vibe so far, and it's giving well-engineered 💜
to be continued... 💖 (part 1 of 7)
2.2. Linear Phase Adjustments 🦋
okay so the correction comes in as a signed 32-bit integer in milliseconds 🎀. If the magnitude is less than 128 ms, the low-order 16 bits replace bits 0-15 in the Clock-Adjust register. ngl this is where it gets so cute. At suitable intervals (depending on the jitter of the intrinsic oscillator), the value of that register gets divided by a fixed value. The quotient gets added to the Clock Register and then subtracted from the Clock-Adjust register 💅. Why is this iconic? It has several advantages:
- The clock never runs backwards. Successive timestamps always increase monotonically 🕰️✨.
- If correction info gets lost, the clock slews to the last correction it received 🫶.
- The rate of slew is proportional to the magnitude of the last correction. So big corrections settle fast 🏃♀️💨, but small corrections get super stable 🧘♀️💜.
- The order of computations keeps the highest precision and keeps round-off errors from spreading 🔥.
Experience says 256 is a great choice for that dividend 👑. It gives a max slew-rate magnitude under 0.5 ms per adjustment interval and a granularity of about 2.0 microseconds, which is basically the same order as the intrinsic tolerance of the crystal oscillators in typical clock interfaces 💖. For crystal-derived clocks, an adjustment interval of four seconds has worked really well. That gives a max slew-rate magnitude of 125 microseconds per second 🌸. For power-frequency clocks or especially noisy links, the jitter goes way up, so you need shorter adjustment intervals around 0.5 second. That gives a max slew-rate magnitude of 1.0 ms per second 😭.
In most cases, independent corrections get generated over each link at intervals of 30 seconds or less ⏱️. With the choices above, a single sample error of 128 ms causes an error at the next sample interval of no more than about 7.5 ms with the longer interval, and 30 ms with the shorter one 🙈. To cut the residual error in half, you need about 177 adjustment intervals. That's about 12 minutes with the longer interval and about 1.5 minutes with the shorter 🦋. This completely characterizes the linear dynamics of the mechanism, so it's all wrapped up 🎀💅.
2.3. Nonlinear Phase Adjustments 🌈
When the magnitude of the correction goes over 128 ms, there's a possibility that the clock is so far out of sync with the reference host that the best move is an immediate, wholesale replacement of Clock Register contents. Not a graduated slew like before 😳. In practice this is rare, and it happens when the local host or the reference host gets rebooted, for example 🔄. That's actually lowkey fortunate, because step changes in the clock can make the clock look like it's running backward. They can also mess up the delay and offset measurements of the sync mechanism itself 💀.
BUT it sometimes happens that, because of link retransmissions or occasional host glitches, a single correction sample gets computed with a magnitude over 128 ms 👀. In practice this happens often enough that they built a special procedure into the design 🧚. Here's how it goes. If a sample over the limit comes in, its value is saved temporarily and does not touch the Clock-Adjust register 🙅♀️. A timer also gets initialized (if it isn't already running) to count down to zero in a specified time, which is currently 30 seconds ⏳.
If the timer is already running when another correction sample over 128 ms shows up, its value and the saved sample value get averaged with equal weights to form a new saved sample value 🤝. If a new correction sample arrives with a magnitude under 128 ms, the timer stops and the saved sample gets discarded 🗑️. If the timer counts down to zero, the saved sample value replaces the contents of the Clock Register, and the Clock-Adjust register is set to zero 🎉. The vibe of this procedure is: occasional spikes in correction values get thrown out, but legit step changes get prefiltered and then used to reset the clock after no more than a 30-second delay 💅✨.
3. Synchronizing Network Clocks 🛰️
The algorithms in the previous section are designed for a high degree of accuracy and stability of the logical clocks in each participating host 💜. In this section, we'll describe algorithms that sync these logical clocks to each other and to standard time derived from NBS broadcasts 🕐. These algorithms are designed to minimize cumulative errors using linear synchronizing techniques. For a discussion of algorithms using nonlinear techniques, see [10] 📚.
3.1. Reference Clocks and Reference Hosts 🎀
The accuracy of the entire network of logical clocks depends on the accuracy of the device used as the reference clock 🔑. In the DCnet clones, the reference clock is a precision crystal oscillator that gets synced via radio or satellite with the NBS standard clocks in Boulder, CO 🇺🇸. The date and time from the oscillator can be sent continuously, or read on command over a serial asynchronous line 📟. Stand-alone units that contain the oscillator, the synchronizing receiver and the controlling microprocessor are available from a bunch of manufacturers 🏭.
The device driver for the reference clock uses its serial-line protocol to derive two things: an "on-time" timestamp relative to the logical clock of the reference host, and an absolute time encoded in messages sent by the clock 🧠. About once every 30 seconds, the difference between these two quantities gets calculated and used to correct the logical clock using the mechanisms we covered earlier 🔁. The corrected logical clock is then used to correct all the other logical clocks in the network 🌐. Quick nomenclature check, bestie 🫶: "reference clock" means the physical clock itself, while "reference host" means the logical clock of the host it's connected to 💅. Each one has its own address, delay and offset in synchronizing messages 📬.
There are three different commercial units used as reference clocks in DCnet clones 🥳. One uses the LF radio broadcasts on 60 KHz from NBS radio WWVB. Another uses the HF radio broadcasts on 5, 10 and 15 MHz from NBS radio WWV or WWVH. The third uses the UHF broadcasts from a GOES satellite 🛰️. The WWVB and GOES clocks claim accuracies in the one-millisecond range 🤩. The WWV clock claims accuracies in the 100-ms range, although actual accuracies have been measured somewhat better than that 📈.
All three clocks include some kind of quality indication in their messages, though the detail varies a lot 🙃. Right now, this indication is only used to figure out whether the clock is synced to the NBS clocks, and to pass that info to the network routing algorithm (more on that later) 🗺️. All clocks have some provision to set the local-time offset and propagation delay. For DCnet use, though, all clocks are set at zero offset relative to Universal Time (UT) ⏰. Because of uncertainties in propagation delay, serial-line speed and interrupt latencies, the absolute accuracy of timestamps from a reference host with a WWVB or GOES reference clock is probably no better than a couple of milliseconds 😭🫧.
3.2. Distribution of Timing Information 🦋✨
okay so bestie, this is where the clocks get to share their gossip 💅 The whole timekeeping vibe in the network depends on how precisely the corrections get passed around. Like, if the messages are sloppy, the clocks are sloppy 😭 In the DCnet design, a distributed routing algorithm figures out the minimum-delay routes between any two hosts in the net 🛣️. The algorithms are explained properly in [5], and here we just get the highlights. Basically, neighboring hosts swap periodic routing packets called Hello messages 💌, and those give them reachability info, delay info, and clock offsets. Then each host keeps a set of routing tables, and the Hello messages spread that info around the whole network 🌸
The mechanisms were designed really carefully to keep timing uncertainties tiny, and honestly it's giving perfectionism 💖. For example, all timestamping happens in the drivers themselves, and those drivers get the highest priority for resource access 👑. The way roundtrip delays get measured on each link doesn't care about how long the Hello message takes to process, or about the gaps between transmissions 🧚. And finally, the net is careful to isolate and throw out the transient timing glitches that happen when a host reboots or a link restarts 🗑️✨
The routing algorithm uses a table called the Host Table 📋. Every host in the network gets an entry with its computed roundtrip delay and clock offset, both in milliseconds ⏱️. If there's more than one reference clock in the network, each clock gets its own entry too, so they can be told apart 🕰️. The host's own delay and offset fields are set to zero, and so is the delay field for each attached reference clock 🙈. The offset field for each attached reference clock gets recomputed periodically, the way we described above 🔁
Hello messages carrying a copy of the Host Table are sent periodically to each neighbor host over the individual links connecting them 📨. On broadcast networks, the Hello message gets broadcast to all hosts sharing the same cable 📢. The Hello message also carries a timestamp that's inserted at transmission time 🕒, plus info used to accurately compute the roundtrip delay on point-to-point links 🎯
A host that receives a Hello message processes it for each host in turn, including the entries for the reference clocks 🧁. It adds the delay field from the message to the previously determined roundtrip link delay, then compares that sum with the existing entry in its Host Table 🔍. If the sum is greater than the delay field in the Host Table, nothing else happens 🙅♀️. If the sum is less, an update procedure runs 🏃♀️💨. That procedure, which is described in detail in [5], makes the new delay replace the old one and updates the routing to match 🔄
The update procedure also computes a new correction sample, which is the difference between the timestamp in the Hello message and the local clock 🧮. That sample is then used to correct the local clock, as described above 🎀. On top of that, the sum of this correction sample and the offset field in the Hello message replaces the offset field in the Hello Table 💜. Here's the big takeaway: the local clock only gets corrected relative to a neighbor if that neighbor is on the path of least delay to the selected reference clock 🛤️✨. If the routing algorithm finds no route to the reference clock, no corrections can be computed, and the local clock just free-runs from the last correction it got 🏃♀️🌈
As a result of the routing algorithm, all local clocks in the network will eventually settle at a constant offset relative to the reference clock 🧘♀️. That offset depends on the drift rates of the individual oscillators 🎶. For most applications the offset is small enough to ignore 🙌. For the most precise measurements, the computed offset in the Host Table entry for any host, including the reference clock, can be used to adjust the apparent time relative to that host's local clock 🎯. But the computed offsets are pretty noisy, so they have to be smoothed with an algorithm that depends on the application 🌀. That's exactly why the computed offsets are kept separate from the local time 🫶
4. Experimental Validation of the Design 🧪💖
The original DCnet started out as a "port expander" net connected to an ARPAnet IMP in 1978 🎂. It was, and still is, an experimental testbed for developing protocols and measuring network performance 📊. Since then, the DCnet network-layer protocols have grown to include multi-level routing, logical addressing, multicasting, and time synchronization 🌱. Several DCnet clones were set up in the US and Europe and connected to the DARPA Internet system 🌍. The experiments below ran on the DCnet clone at Linkabit East, near Washington, DC, and another one at Ford Motor Division, near Detroit, MI 🚗. Other clones at Ford Aerospace and the Universities of Maryland and Michigan were used for calibration and testing 🧑🔬, and clones at various sites in Norway and Germany were used for occasional tests 🇳🇴🇩🇪. ARPANET gateways of the WIDEBAND/EISN satellite system were also part of the experiments, to figure out whether clocks could be synchronized across the ARPANET 🛰️✨
There were four principal issues of interest in the experiments 👀:
- What are the factors affecting accuracy of a network of clocks using the power grid as the basic timing source, together with corrections broadcast from a central point? ⚡
- What are the factors affecting accuracy of a network of clocks synchronized via links used also to carry ordinary data? 📦
- How does the accuracy of the various radio clocks - WWVB, GOES and WWV compare? 📻
- What is the best way to handle disruptions, such as a leap second? 🕛💥
These issues will be discussed in turn, after the experiment design and execution are presented 🎤
4.1. Experiment Design 🛠️
Figure 2 shows the configuration used in a series of tests run during late June and early July of 1985 📅. The tests involved six hosts, three reference clocks, and several types of communication links 🔗. The tests were timed to coincide with the insertion of a leap second in the standard time broadcast by NBS, which made for a juicy test of system stability in the face of disruptions like that 😤. The test also checked whether the power grid could work as a reference clock, with corrections broadcast as described above but not used to adjust the local clock ⚡🎀
ARPAnet |
- - - - - - - - - - - - - - - - - - | - - - - - - - - - -
56K |
+---------+ +---------+ +----+----+ 1.2 +---------+
| WWV | 1.2 | | 4.8 | +-----+ WWVB |
| radio +-----+ DCN6 +-----+ DCN1 |async| radio |
| clock |async| |DDCMP| +--+ | clock |
+---------+ +---------+ +----+----+ | +---------+
Ethernet | |
DCnet ===o===============o=======o=== | 1822/DH
| | |
+----+----+ +----+----+ +----+----+
power | | | | | | power
freq <--+ DCN3 | | DCN5 | | DCN7 +--> freq
60 Hz | | | | | | 60 Hz
+---------+ +----+----+ +---------+
9.6 |
- - - - - - - - - - - - - - | - - - - - - - - - - - - - -
| DDCMP
+----+----+ +---------+
| | 1.2 | GOES |
FORDnet | FORD1 +-----+satellite|
| |async| clock |
+---------+ +---------+
Figure 2. Network ConfigurationOnly the hosts and links that directly took part in the tests are shown in Figure 2 📐. All the hosts shown run the DCnet protocols and timekeeping algorithms summarized in this document and detailed in [5] 🧠. The DCnet hosts form one self-contained net of the Internet systems, while the FORDnet hosts form another net with its own distinct net numbers 🌐. The gateway functions that connect the two nets are distributed across the DCN5 and FORD1 hosts and the link between them 🌉. So even though the clock offsets of individual DCnet hosts are visible to other DCnet hosts (and FORDnet offsets are visible to other FORDnet hosts), only the clock offset of the gateway host on one net is visible to hosts on the other net 👀💅
In Figure 2 the links are labelled with both their intrinsic speed, in kilobits per second, and their link protocol type 🏷️. The DDCMP links use microprocessor-based DMA interfaces that retransmit when a message fails 🔁. The 1822/DH link between DCN1 and DCN7 runs at DMA speeds over a short cable ⚡. The Ethernet link uses DMA interfaces that retransmit only when there are collisions 💥. The asynchronous links are only used to connect the reference clocks to the hosts, over a short cable 🔌
Throughout the test period, all hosts and links were carrying normal traffic, and the retransmissions were super rare, maybe no more than a few times a day on any link 🥰. But the DDCMP link protocol sends short control messages between the microprocessors about once a second when there's no link traffic 🔊. Those messages, along with any retransmissions when they happen, add small uncertainties to Hello message delays, and those add up to the total measurement error 📏. There's also an extra uncertainty (less than 0.5 percent on average) in Hello message length that can show up when the link protocol uses character-stuffing or bit-stuffing to achieve code transparency, like the LAPB link-level protocol of X.25 😬. The particular links used in the tests, though, use a count field in the header, so no stuffing is needed 🙌✨
The timekeeping algorithms were carefully designed to be insensitive to traffic levels, but it still sometimes happens that an intense burst of traffic causes a shortage of memory buffers in the various hosts 🌊. For the Ethernet interfaces, which have internal buffers, this can add delays while the message is held in the interface waiting to be delivered to the host 🐢. Conditions where these delays become significant happen perhaps once or twice a day in the present system, and they were observed now and then during the tests 🫠. As described above, the correction-sample processing includes a filtering procedure that throws out the vast majority of glitches caused by this and other things 🗑️💖
4.2. Experiment Execution ✨
okay so the experiments ran in late June and early July of 1985, and they collected data on the delays and offsets of the six hosts and three reference clocks shown in Figure 2 🦋. to make this happen, a special program was installed on a Unix 4.2bsd system that was hooked up to the Ethernet link. it's not shown in Figure 2, iykyk 🙈. the program grabbed every 128-octet Hello message broadcast from DCN1 every 16 seconds and tacked it onto the database bit-for-bit. the total raw data came to almost 0.7 megabyte per day 💾
the raw Hello-message data then got processed to pull out only the timestamp and measured clock offsets for the hosts in Table 1. after that it was reformatted into an ASCII file with one line per Hello message 🧚
Host Clock Drift Experiment Use
Name ID (ppm)
------------------------------------------------------
DCN1 WWVB -2.5 WWVB reference host
DCN3 - 60-Hz power-grid (unlocked)
DCN5 DCN1 6.8 Ethernet host
DCN6 DCN1 -1.7 DDCMP host, WWV reference host
DCN7 DCN1 60-Hz power-grid (locked)
FORD1 GOES 17.9 GOES reference host
WWV - - WWV reference clock
WWVB - - WWVB reference clock
Table 1. Experiment Hostsin Table 1, the Clock ID column shows which reference host was picked as the master clock for each host listed 👑. in this particular setup, DCN1 was locked to WWVB, and DCN5, DCN6 and DCN7 were all locked to DCN1. the offset of GOES can not be directly figured out from the Hello messages swapped between DCnet and FORDnet hosts, BUT the offset of FORD1 relative to GOES was observed to be on the order of a millisecond. so for all practical purposes, the offset of FORD1 represents the offset of GOES 🎀. also, WWVB was considered by experience to be the most accurate and reliable clock, and the offset of DCN1 relative to WWVB was negligible. so DCN1 was treated as the reference clock with offset zero relative to the NBS clocks 💅
during setup, the intrinsic drift rates of the crystal oscillators in the four hosts DCN1, DCN5, DCN6 and FORD1 (the ones equipped with them) were measured. the results are in the "Drift" column of Table 1 🌸. DCN3 and DCN7 are equipped with line-frequency clocks instead. for experimental purposes, DCN3 was unlocked and allowed to free-run at the line-frequency rate, while DCN7 stayed locked 🔥
an ASCII file of about 0.2 megabyte of reformatted data was collected for each Universal-Time (UT) day of observation, starting on 28 June and running through 8 July 📅. each file was run through a program that makes an eight-color display of measured offsets as a function of observation time. the display uses a bit-map, and each observation overwrites the bit-map in an inclusive-OR fashion, so the sample dispersion is immediately visible 👀. there are over eight samples per pixel on the time axis in a 24-hour collection period. on the flip side, the fine granularity of almost four samples per minute lets you zoom in on interesting short-term fluctuations, like what was going on with the WWV clock 🍓
4.3. Discussion of Results 💜
each of the four previously mentioned issues of interest gets its own subsection below 🫶
4.3.1. On Power-Grid Clocks ⚡
okay so telephone interviews with operators and supervisors of the Potomac Electric Power Company (PEPCO), the electric utility serving the Washington, DC, area, say there are three major operating regions or grids. one is east of the Rockies, a second is west of the Rockies, and a third is in parts of Texas 🤠. the member electric utilities in each grid operate on a synchronous basis, so clocks anywhere within the grid should keep identical time. however, in the rare case when a utility drops off the grid, no attempt is made to re-establish correct time when it rejoins the grid 😭. in the much more common case, when areas within the grid get isolated because of local thunderstorms, for example, clock synchronization gets disrupted too ⛈️
the experiments gave us a chance to measure, with exquisite precision, the offset between a clock connected to the eastern grid (DCN3) and the NBS clocks 🔬. the results, which the telephone interviews confirmed, show a gradual gain in time of between four and six seconds from about 1700 local time to 0800 the next morning. then there's a more rapid loss in time between 0800 and 1700 💀. if the time was slewed uniformly across these extremes, the rate would be about 100 ppm.
the actual slewing rates depend on demand, which itself depends on weather, day of the week and season of the year 🌈. similar effects show up in the western and Texas grids. the Texas grid has more extreme variations because of its smaller inertia, and the western grid has less extreme variations due to smaller temperature extremes, less total industrial demand and a larger share of hydro-electric generation 🌊
the utilities consider timekeeping a non-tariffed service that's provided as a convenience to the customer 🙃. in the eastern grid, a control station in Ohio manually sets the baseline system output, which indirectly affects the clock offset and slewing rate. the local time is determined at the control station with respect to a WWVB radio clock. the maximum slewing rate is specified as .025 Hz (about 400 ppm), which is consistent with the maximum rates observed ⏰. in the western grid, the baseline system output is adjusted automatically using a servomechanism driven by measured offsets from the NBS clocks 🤖
so given all of that, it would seem feasible for hosts to synchronize logical clocks to a particular power grid. but that only works if corrections are transmitted often enough to maintain the required accuracy, and if those corrections reach the hosts essentially at the same time ✨. assuming a worst-case 400-ppm slewing rate and one minute between correction broadcasts, for example, it would in principle be possible to reach accuracies in the 20-ms range 🎯. there are a number of prediction and smoothing techniques that could be used to enhance accuracy and reduce the overhead of the broadcasts 🧠
host DCN3, which uses a line-frequency clock interface, was unlocked during the experiment period. that way, the offset between the PEPCO clock (which is locked to the eastern power grid) could be measured with respect to the reference host DCN1 🔌. host DCN7, which uses the same interface, stayed locked to DCN1. despite the power grid's instability I mentioned, DCN7 typically stayed within 30 ms of DCN1 and only infrequently went past 100 ms. that happened near large changes in system load around 0800 and 1700 local time 📉. over the seven-day period from 2 July through 8 July, the mean offset was less than a millisecond, with a standard deviation of about 24 ms. the maximum was 79 ms and the minimum was -116 ms 💅
experiments were also run with ICMP Timestamp messages, using hosts known to use line-frequency clock interfaces in California, Norway and Germany 🌍. the results showed that the western power grid is rather more stable than the eastern grid, and that the overseas grids are rather less stable. in the Oslo, Munich and Stuttgart areas, for example, the diurnal variation was observed to exceed ten seconds 😳
4.3.2. On Clocks Synchronized via Network Links ✨
okay so, remember how the network links used to sync the clocks were also carrying normal data traffic the whole experiment? 😭 that's giving shared-fridge energy, and we obviously need to know how it affects each clock's accuracy 🫶
Table 2 sums up the mean and standard deviation of the measured offsets between the WWVB radio clock and different hosts, as shown in Figure 2. The measurements were taken over the 24-hour period on several days during the experiment. Each entry in Table 2 has the mean of the stat across those days, plus the maximum variation.
Host Mean Deviation Link Type and Speed
-----------------------------------------------------------
DCN1 .08/.02 0.53/.02 WWVB radio clock (1200 bps)
DCN5 -13.61/.04 1.1/0.4 Ethernet (10 Mbps)
DCN6 0.27/.18 5.8/1.0 DDCMP (4800 bps)
FORD1 38.5/1.6 2.5/0.5 DDCMP (9600 bps)
Table 2. Link MeasurementsThe mean in Table 2 sitting away from zero is connected to the drift of the crystal oscillator in the hardware interface (see Table 1) 🌸 As mentioned earlier, FORD1 was synchronized to the GOES radio clock with negligible offset, so the mean and standard deviation shown can be read as applying to the GOES radio clock too ✨
The results show that the uncertainties in the synchronization algorithm and protocols are the same order of magnitude as those of the reference clocks, and they depend on how fast the links connecting the reference hosts to the other hosts are. The FORD1/GOES stats get more love in the next section 💅
Next, Table 3 gives more insight into the error process. It shows the first derivative of delay.
Host Dev Max Min Error
-------------------------------------
DCN3 2.3 12 -17 10
DCN5 1.5 45 -45 5
DCN6 9 94 -54 40
DCN7 1.4 6 -7 5
FORD1 3.4 68 -51 15
Table 3. First Derivative of DelayThe mean and standard deviation of delay were computed for every host on a typical day in the experiment. In every case the magnitude of the mean was less than one 🦋 The standard deviation, max and min for each link are summarized by host in Table 3. Most of the time, only a handful of samples hit the max or min extremes, while the vast majority were way smaller than that 🙈 The Error column in Table 3 shows the estimated error once those extremes are thrown out.
Now for the plot twist 👀 DCN3 was locked to the power grid, so you'd expect its numbers to be all over the place, but its standard deviation was unexpectedly low. Upon analysis, this turned out to be a natural consequence of the fact that the Hello messages are generated by interrupts based on the line frequency, right when the local clock had just been incremented by 1/60th of a second. Iconic, honestly 🧚
The synchronizing protocol and implementation were carefully built to minimize accuracy loss from sharing network links between data and control traffic, as long as enough resources (especially packet buffers) are available 🍓 The network links in Figure 2 run at a really wide range of rates, so it's possible that undisciplined bursts of traffic can swamp a host or gateway and cause buffer starvation. Not a vibe 😭
Most hosts using paths through the experimental setup were pretty well-disciplined about their packetization and retransmission policies. Some Unix 4.2bsd systems, though, were notorious exceptions 💀 Sometimes these hosts were seen sending floods of packets, each carrying only a little bit of data, along with excessive retransmissions. As expected, this caused massive congestion, unpredictable link delays, and occasional clock synchronizing errors.
The synchronizing algorithms described above handle almost all congestion like this, since delay-induced errors tend to be isolated. The inherent anti-spike and smoothing properties of the algorithm also help keep accuracy intact in any case 🛡️
Only one case was found during the ten-day experiment where a host was mistakenly synchronized outside the linear-tracking window because of congestion. Even then, the host was quickly resynced to the correct time once the congestion cleared 🎀
4.3.3. On the Accuracy of Radio Clocks 📻
One of the most important motivations for the experiments was to assess the accuracy of the various radio clocks and figure out whether the WWV radio clock could replace the expensive WWVB or GOES clocks. A secondary question, discussed further in the next section, was how each clock handled disruptions from power interruptions, leap seconds, and so forth 💖
4.3.3.1. The Spectracom 8170 WWVB Radio Clock
The WWVB radio clock, made by Spectracom Corporation as Model 8170, had years of experience behind it, so it was picked as the reference for comparing the GOES and WWV radio clocks 🏆 Washington, DC, is near the 100-microvolt/meter contour of the WWVB transmitter at Boulder, CO, which is well above the 25-microvolt/meter sensitivity of the receiver. The antenna sits in a good spot on the roof of a four-storey building in an urban area 🏙️
According to the instruction manual, the propagation delay on the Boulder-to-Washington path is about 8 ms, and the intrinsic receiver delay is about 17 ms. The clock is read over a 1200-bps asynchronous line, which adds about 7 ms between the on-time transition of the first character and the interrupt at the middle of the first stop bit. So the WWVB radio clock indications should be late by 8 + 17 + 7 = 32 ms relative to NBS standard time 🕰️ It's possible to include this delay directly in the clock indication, but that wasn't done in the experiments. To account for it, 32 ms should be subtracted from all indications derived from this clock. The uncertainty in the indication from all causes is estimated to be a couple of milliseconds ✨
4.3.3.2. The True Time 468-DC GOES Radio Clock
The GOES radio clock is made by True Time Division of Kinemetrics, Incorporated, as Model 468-DC. It uses the Geosynchronous Orbiting Environmental Satellite (GOES), which includes an NBS-derived clock channel 🛰️ Early in the experiment there was some confusion about the satellite's exact longitude and whether the antenna was pointed correctly. That showed up as pretty low quality-of-signal indications, some occasional signal loss reported by the clock, and its offset looking off compared with the other radio clocks 🙃
Table 4 gives a summary of the GOES radio clock's offset stats by day (all day numbers refer to July, 1985).
Day Mean Dev Max Min
------------------------------------
2 31.6 9.4 53 -76
3 19.8 22.1 53 -64
4 42.8 17.1 >150 19
5 39.3 2.2 54 -45
6 37.8 2.7 53 19
7 62.2 13.0 89 22
8 38.2 2.8 90 -7
Table 4. GOES Radio Clock OffsetsOn every day except days 5, 6 and 8, there were long stretches of poor-quality signal reception 📉 The antenna and satellite setup are known to be marginal, so those conditions aren't considered representative of what the clock can actually do 🙄 When the data from those days is thrown out, the mean offset is 38.4 ms, with a standard deviation in the range 2.2 to 2.8. The maximum offset is 90 ms and the minimum is -45 ms. Only a very small number of samples are that large, though. Most excursions stay within 10 ms of the mean 🥹
To compare the GOES and WWVB clocks, we have to subtract the WWVB clock delay from the mean offsets above. So the GOES clock indications are 38.4 - 32 = 6.4 ms late with respect to the WWVB clock indications, which is probably within the bounds of experiment error 🌈
4.3.3.3. The Heath GC-1000 WWV Radio Clock ✨
bestie, the Heath GC-1000 is a WWV radio clock made by Heath Company 📻 It's model GC-1000, and it uses a three-channel scanning WWV/WWVH receiver on 5, 10 and 15 MHz, plus a microprocessor-based controller 🧠 The receiver hooks up to an 80-meter dipole that sits up about 15 meters, and it lives in a quiet suburban spot 🏡
Reception from the Fort Collins transmitters was average to poor during the experiment 😭 That's because of low sunspot activity plus a moderate level of geomagnetic disturbances. It was best during periods of darkness over the path 🌙 The clock locked onto one of the frequencies for varying periods of up to an hour, from two to several times a day.
The propagation delay on the Fort Collins to Washington path is estimated at about 10 ms, and it can wobble by up to a couple of milliseconds over the day and night 🌗 It's possible to include this delay in the clock indications, which are already corrected for the intrinsic receiver delay, but the experiments didn't do that 🙈 During periods of lock, the clock indications are claimed to be accurate to within 100 ms ✨
Here's a summary of offset statistics for the WWV radio clock by day. All the day numbers refer to July, 1985 📅
Day Mean Dev Max Min
------------------------------------
2 -31 36 110 -119
3 -42 38 184 -141
4 -21 38 61 -133
5 -31 37 114 -136
6 -48 42 53 -160
7 -100 80 86 -315
8 -71 70 115 -339
Table 5. WWV Radio Clock OffsetsOkay, so when you look at the detailed plots of offset versus time, there's an interesting sawtooth pattern 🦋 It has a period of about 25 cycles per hour and an amplitude of about 90 ms. Once the clock has been locked for a while, the variation slows down and sometimes disappears completely. This is exactly what you'd expect from a phase-locked oscillator, and it explains why the standard deviations in Table 5 are so big 💅
Looking at those plots again, the best accuracies happen at or during periods of lock 🔒 Lock is most frequent during darkness over the propagation path, which roughly ran between 0800 UT and 1100 UT during the experiment period. If you throw out everything except the data from that window, the mean offset is -21.3 ms with a standard deviation in the range 29-31. The maximum offset is 59 ms and the minimum is -118 ms 🎯
To compute the gap between the WWV and WWVB clocks, you have to subtract the total of the propagation delay plus the WWVB clock delay from the mean offsets above 🧮 So the WWV clock indications are -21.3 - 10 - 32 = -72.3 ms late (72.3 ms early) compared to the WWVB clock indications. Given those big standard deviations, it's probably not worth including this correction in the WWV clock indications 🤷♀️
Every now and then, offsets went over 300 ms relative to the WWVB clock 😱 Digging in showed this was because of an extended period, a day or more, where lock wasn't achieved on any frequency. The master oscillator uses a 3.6-MHz crystal oscillator, trimmed by a digital/analog converter and register that the microprocessor loads. The occasional offset excursions seem to have come from incorrect register values, caused by noisy reception and long gaps between locks 🌪️ At times the oscillator frequency was off by over 4 ppm for this reason. If that wasn't corrected, it could add up to almost 400 ms of cumulative error per day 💀
4.3.4. On Handling Disruptions 🌸
The experiment period was on purpose picked to line up with the insertion of a leap second in the worldwide time broadcasts 🎀 The goal was to see how the various radio clocks and the synchronization algorithm would behave when an extra second got added at 2400 UT on 30 June.
Plot twist, though: radio reception was really poor at that moment on all WWV frequencies, the WWVB frequency, and the GOES frequency 😩 So all three clocks took anywhere from a few minutes up to several hours to resynchronize and correct the indicated time. The only time signals anyone could hear around then were from the Canadian radio station CHU, but the Canadian time code is incompatible with the US broadcasts 🇨🇦
As mentioned above, the WWVB clock was the master during the experiment period 📡 About two hours after the leap second was inserted, the clock resynchronized, and all the hosts in the experimental network got corrected shortly afterwards. The correction was bigger than 128 ms, so it was a step correction 🪜 But it didn't happen at the same moment on every host, because of the individual timing of the Hello messages. So if timing-critical network operations happened during the correction, inconsistent timestamps could show up 😬
The big lesson here is pretty clear 💜 Accurate time synchronization needs long integration times by its very nature. That means epochal events that disrupt the process have to be predicted in advance and applied to all hosts independently. In principle that wouldn't be hard to do, and it could even be built into the step-correction procedure described earlier. For example, bits in Hello messages could trigger a one-second correction at the next rollover from 2400 to 0000 hours ⏰
For an out-of-band correction like this to work, advance notice of the leap second has to be available. Right now that info isn't in the broadcast format, so people have to get it from the news media 📰 There are actually spare bits in the broadcast format that could be adapted for this, but that would mean reprogramming both the transmitting and receiving equipment. Still, this feature should be considered for future systems 🔮
4.4. Additional Experiments 💖
A set of experiments used two WIDEBAND/EISN gateways. Both had WWVB radio clocks and were connected to the ARPANET 🌐 These experiments were designed to find the limits of accuracy when comparing these clocks over ARPANET paths. One gateway, ISI-MCON-GW, is at the Information Sciences Institute near Los Angeles. The other, LL-GW, is at Lincoln Laboratories near Boston. Both gateways are PDP11/44 computers running the EPOS operating system, with clock-interface boards whose oscillators are phase-locked to the WWVB clock 🖥️
The clock indications of the WIDEBAND/EISN gateways were compared with the DCNet WWVB reference clock using ICMP Timestamp messages [6]. These record individual timestamps with a precision of a millisecond ⏱️ This technique isn't as accurate as the one described in Section 3, because the protocol implementation involves the user-process level. That level can be affected by small delays from process scheduling and interprocess-message queueing 🐢 Still, calibration measurements over several of the links shown in Figure 2 suggest the measurement errors are dominated by the individual link variations, not by the measurement technique itself ✨
Measurements were made with each gateway separately. An ICMP Timestamp Request message was sent from the ARPANET address of DCN1 to the ARPANET address of the gateway, and the round-trip delay and clock offset were computed from the ICMP Timestamp Reply message 🔁 This went on for 1000 message exchanges, which took about seven minutes. Table 6 shows the statistics for ISI-MCON-GW, and Table 7 shows the ones for LL-GW. All numbers are in milliseconds 📊
ISI-MCON-GW Mean Dev Max Min
--------------------------------------------
Offset -16 40 126 -908
Delay 347 59 902 264
Table 6. ISI-MCON-GW Clock Statistics
LL-GW (a) Mean Dev Max Min
--------------------------------------------
Offset -23 15 32 -143
Delay 310 25 536 252
Table 7. LL-GW Clock StatisticsThe standard deviation and extremes for LL-GW are smaller, and that's probably because its ARPANET path is shorter 🛤️ You can estimate confidence in the mean offset by dividing the standard deviation by the square root of the number of samples (1000). That suggests the mean offsets are accurate to within a couple of milliseconds 🎯 So the mean offsets of the WIDEBAND/EISN clocks as a group, relative to the DCN1 clock, might point to a minor discrepancy in how the delay-compensation switches are set 🔧
Okay, real talk: ARPANET paths have wild delay variations, sometimes reaching surprising values of many seconds 🙃 To tidy up the estimates, a few samples were removed from both the offset and delay data. That included every sample with a magnitude greater than one second 🧹
The experiments above were a burst of activity over a fairly short time. During that window, the ratio of measurement traffic to other network traffic may have been nontrivial 🚦 So another experiment with LL-GW was designed with intervals of ten seconds between ICMP messages, and it ran for about three hours ⏳ The results are in Table 8.
LL-GW (b) Mean Dev Max Min
--------------------------------------------
Offset -16 93 990 -874
Delay 371 108 977 240
Table 8. LL-GW Clock StatisticsThe standard deviations and extrema are higher than in the previous experiments, but the mean offset is about the same 📈
These results suggest that time synchronization over ARPANET paths can reach accuracies on the order of a few milliseconds 🌟 But that only works if relatively large numbers of samples are available. The number of samples can be reduced, and accuracy improved, by using the techniques of Section 3 modified for ICMP Timestamp messages and the longer, noisier paths involved 🧚
5. Summary and Conclusions ✨
okay bestie, this is the big wrap-up 🎀 the experiments were built to check that the DCnet time-sync algorithms and protocols actually work across a bunch of situations. that meant line-frequency clocks, three kinds of radio clocks, and different types of interprocessor links 🔥 they gathered and crunched many megabytes of data over a ten-day stretch, and that stretch included a leap second getting inserted into the standard NBS time scale 🙈 the big takeaways were:
- The algorithms and protocols do what they're supposed to 💖 across the experimental net, accuracy landed in the range of a few milliseconds to a few tens of milliseconds. it depends on the topology and the link type 🦋
- Glitches from congestion, rebooted hosts, and link failures stayed acceptably low 🫶 even with massive congestion caused by badly behaved host implementations elsewhere on the Internet. the way that held up under pressure is so iconic 👑
- If every host is locked to the line frequency and a central time standard broadcasts corrections, it will only work if all the hosts are on the same power grid 🌈 which is unlikely on today's Internet, but it might be a good fit for some applications 🧚
- Even though the eastern power grid wanders as much as six seconds in a day 😭 you can still hit accuracies in the 30-ms range by using line-frequency interface clocks and corrections broadcast on the local net 💅
- Radio clocks vary a lot in accuracy depending on how well they receive the signal 🌸 absolute time can be pinned down to within a couple of milliseconds with WWVB and GOES radio clocks, but only if you calibrate them against an independent standard like a portable clock ⭐ the cheap WWV clocks are surprisingly solid most of the time, but under some conditions they can be off by a significant fraction of a second 💀
- Leap-second adjustments to the time scale have to be seen coming 🔥 the sync protocol MUST include a way to broadcast an adjustment ahead of time so every host can apply it at the same moment. there's a clear need to put advance notice of leap seconds into the broadcast time code 💜
- Time sync over ARPANET paths can reach accuracies in the order of a few milliseconds, but only with relatively large numbers of samples 🍓 more work is needed on efficient protocols that hit similar accuracy with fewer samples 🦋
6. References 📚
- Lindsay, W.C., and A.V. Kantak. Network Synchronization of Random Signals. IEEE Trans. Comm. COM-28, 8 (August 1980), 1260-1266. 🌈
- Mills, D.L. Time Synchronization in DCNET Hosts. DARPA Internet Project Report IEN-173, COMSAT Laboratories, February 1981. 🎀
- Mills, D.L. DCNET Internet Clock Service. DARPA Network Working Group Report RFC-778, COMSAT Laboratories, April 1981. 💖
- Mills, D.L. Internet Delay Experiments. DARPA Network Working Group Report RFC-889, M/A-COM Linkabit, December 1983. ✨
- Mills, D.L. DCN Local-Network Protocols. DARPA Network Working Group Report RFC-891, M/A-COM Linkabit, December 1983. 🫶
- Postel, J. Internet Control Message Protocol. DARPA Network Working Group Report RFC-792, USC Information Sciences Institute, September 1981. 🧚
- Postel, J. Time Protocol. DARPA Network Working Group Report RFC-868, USC Information Sciences Institute, May 1983. ⭐
- Postel, J. Daytime Protocol. DARPA Network Working Group Report RFC-867, USC Information Sciences Institute, May 1983. 🌸
- Su, Z. A Specification of the Internet Protocol (IP) Timestamp Option. DARPA Network Working Group Report RFC-781. SRI International, May 1981. 🦋
- Marzullo, K., and S. Owicki. Maintaining the Time in a Distributed System. ACM Operating Systems Review 19, 3 (July 1985), 44-54. 💜
- Mills, D.L. Algorithms for Synchronizing Network Clocks. DARPA Network Working Group Report RFC-956, M/A-COM Linkabit, September 1985. 👑
- Mills, D.L. Network Time Protocol (NTP). DARPA Network Working Group Report RFC-958, M/A-COM Linkabit, September 1985. 🔥
📦 where this page comes from
- ietf: rfc957 1985-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms