RFC 1589
A Kernel Model for Precision Timekeeping
RFC 1589: A Kernel Model for Precision Timekeeping 🦋⏰
Bestie, this one is a March 1994 Informational RFC by D. Mills at the University of Delaware 🎀 It is not a standard, just a model people can play with. The title block says it's Category: Informational, which means it's giving "here's an idea, please improve it" energy. ✨ It doesn't obsolete or update any other RFC, so it's a totally standalone vibe. 💅
Status of this Memo 📜
This memo provides information for the Internet community 🌸 It does not specify an Internet standard of any kind, so don't go treating it like one. 🙈 Distribution of this memo is unlimited, so share it with everyone you know. 💜
Overview 🔥
Okay so this memo describes an engineering model that gives a generic operating system a precision time-of-day function ⏰ It's built on the ideas of disciplined oscillators and phase-lock loops (PLL), which show up a lot in the engineering literature. 🤓 The model has already been implemented in the Unix kernel for several workstations, including ones made by Sun Microsystems and Digital Equipment. 💖
The model changes how the system clock gets adjusted in both time and frequency 🧚 It also gives you ways to discipline the frequency against an external precision timing source. 🦋 It includes a generic system-call interface meant for the Network Time Protocol (NTP) or a similar time synchronization protocol. ✨ The NTP Version 3 daemon xntpd runs on top of this model, and the synchronization it gives you is limited, in principle, only by how accurate and stable the external timing source is. 👑
This memo doesn't obsolete or update any RFC 🫶 It also doesn't propose a standard protocol, specification or algorithm. It's meant to provoke comments, refinement and alternative implementations, so bring your ideas! 💅 You don't need to know NTP to get the design principles or the implementation 🌈 but it does help when you want to see how the model acts inside a fully working timekeeping system. 🙈 The architecture and design of NTP are described in [1]. The current NTP Version 3 protocol specification is in RFC-1305 [2], and a subset of the protocol, the Simple Network Time Protocol (SNTP), is in RFC-1361 [4]. 🍓
The model has been implemented in three Unix kernels, for Sun Microsystems and Digital Equipment workstations 🔥 For the Digital machines, the model also gives you improved precision down to one microsecond (us) ⭐ These implementations touch licensed code, so the author can't hand them out directly, and inquiries should go to the manufacturer's representatives. 😭 The good news is the engineering model itself, which includes a simulator with code segments that are almost identical to the implementations but don't involve licensed code, is free to grab via anonymous FTP from host louie.udel.edu, in the directory pub/ntp, as the compressed tar archive kernel.tar.Z. 🎀 The NTP Version 3 distribution is on the same host and directory as the compressed tar archive xntp3.3g.tar.Z. The version number shown, 3.3g, may change as new versions come out. 🌸
1. Introduction ✨
Okay bestie, this section is the "here's what this whole thing is for" part. 💖 The memo describes a model and a programming interface for generic operating system software that manages the system clock and timer functions 🧚 The model gives most workstations and servers better accuracy and stability when they use NTP or a similar time synchronization protocol. 🔥 It explains the principles behind designing and implementing the model, and related technical reports go deeper into the design approach, the engineering analysis and the performance evaluation of the Unix kernels for Sun Microsystems and Digital Equipment workstations. 🤓
The NTP Version 3 daemon xntpd works with these implementations to give you better accuracy and stability, plus less overhead in the operating system and on the network. ✨ The model also supports external timing sources, like precision pulse-per-second (PPS) signals and the industry standard IRIG timing signals. 🦋 The NTP daemon detects the new features automatically and uses them whenever they're available. 👑
There are three prototype implementations of the model in this memo: one for the Sun Microsystems SPARCstation running the SunOS 4.1.x kernel, one for the Digital Equipment DECstation 5000 running the Ultrix 4.x kernel, and one for the Digital Equipment 3000 AXP Alpha running the OSF/1 V1.x kernel. 🌈 For the DECstation 5000/240 and the 3000 AXP Alpha machines, a special feature gives improved precision down to 1 us. (Sun 4.1.x kernels already give 1-us precision, so they're already iconic on that front.) 💅
Besides improving the system clock's accuracy, stability and precision, these implementations don't change how existing Unix system calls that manage the system clock work. That includes gettimeofday(), settimeofday() and adjtime() 🫶 If you turn on the new features, though, the operations of gettimeofday() and adjtime() can be controlled instead by the new system calls ntp_gettime() and ntp_adjtime(), which are covered below. 🦋
The memo gives a detailed description of the variables and algorithms, in the hope that someone will be able to build similar functionality into Unix kernels for other machines 🧚 The algorithms only need minor changes to the system clock and interval timer routines. They also include interfaces so application programs can read the system clock status and some statistics about the time synchronization process. 🔥 Detailed installation instructions are in the specific README files that come with the kernel distributions. 🍓
In this memo, NTP Version 3 and the Unix implementation xntp3 are used as an example application of the new system calls, for use by a synchronization daemon. 💖 In principle, other protocols and implementations can use the new system calls too 😭 Even when local time is kept up to date with periodic message exchanges at fairly long intervals, like with the NIST Automated Computer Time Service, being able to adjust the system clock frequency precisely makes the synchronization procedures simpler. It also lets you cut the telephone call frequency quite a bit. 📞✨
2. Design Approach 🎀
Okay, this part explains how NTP controls the system clock on a client workstation. 🤓 It's not strictly necessary for understanding or implementing the model, but it's helpful. 🫶 As described in [1], NTP exchanges timestamps with one or more peers that share a synchronization subnet, so it can calculate the time offsets between peer clocks and the local clock. 🦋 Several algorithms then process those offsets, refining and combining them into an ensemble average. That average is used to adjust the local clock's time and frequency. 🔥 How the local clock gets adjusted is the main topic of this memo, and the goal is the most accurate and stable system clock possible, given the kernel software and workstation hardware you have. 👑
To get how the new software works, it helps to see how most Unix kernels keep track of system time ⏰ In Unix, a hardware counter interrupts the kernel at a fixed rate: 100 Hz in the SunOS kernel, 256 Hz in the Ultrix kernel and 1024 Hz in the OSF/1 kernel. 🌈 The Ultrix timer interval (the reciprocal of the rate) doesn't divide one second evenly into microseconds, so the Ultrix kernel adds 64 microseconds once each second. That means the timescale is made of 255 advances of 3906 us plus one advance of 3970 us. 🤯 The OSF/1 kernel is similar: it adds 576 us once each second, so its timescale is 1023 advances of 976 us plus one advance of 1552 us. 💀
2.1. Mechanisms to Adjust Time and Frequency 🧚
Most Unix kernels let you slew the system clock to a new offset relative to the current time using the adjtime() system call ✨ To do this, the clock frequency is changed by adding or subtracting a fixed amount (tickadj) at each timer interrupt (tick), for a calculated number of ticks. 🔥 That calculation divides the requested offset by tickadj, so you can only slew to a new offset with a precision of tickadj, which is usually around 5 us, but sometimes a lot more. 😭 This creates a roundoff error that can pile up to an unacceptable degree, so the clock adjustment procedures of the synchronization daemon need special provisions. 🫶
To implement a frequency-discipline function, you have to give the kernel time offset adjustments at regular adjustment intervals using adjtime(). 🦋 To get the system clock jitter down to the level this memo is aiming for, the adjustment interval needs to be fairly small, around 1 s. 💖 The catch is that the Unix adjtime() implementation needs each offset adjustment to finish before another one can start. So large adjustments have to be spread out across possibly many adjustment intervals. 🌸 Handling the adjustment interval and compensating for roundoff error makes the synchronizing daemon a lot more complicated. 🙈
The new model swaps this scheme out for one that represents the system clock as a multiple-word, precision-time variable, which allows very precise clock adjustments 💅 At each timer interrupt, a precisely calibrated quantity is added to the kernel time variable, and overflows are propagated as required. 🔥 That quantity is computed the same way as in the NTP local clock model described in [3], which works as an adaptive-parameter, first-order, type-II phase-lock loop (PLL). 🤓 In principle, this PLL design can control the system clock oscillator to within 1 us, and frequency to within parts in 10^11. 🌈 Those levels of precision are way beyond what the CPU clock oscillator in typical workstations can do, but they're right on target when you use the precision external oscillators described below. 👑
The PLL design is identical to the one first implemented in NTP and described in [3]. 🧚 In this design, the software daemon simulates the PLL using the adjtime() system call, but as we covered above, that makes the daemon a lot more complicated. ⏰ The modified kernel routines implement the PLL inside the kernel, using precision time and frequency representations, so all of those complications go away! 🎀 A new system call, ntp_adjtime(), is only called as each new time update is determined. In NTP, that happens at intervals of anywhere from 16 s to 1024 s. 🦋 Doing the frequency compensation in the kernel also means the system time keeps running true even if the daemon stops working or the network paths to the primary synchronization source fail. 🫶 Main character energy for the kernel, honestly. 💜
In the new model, ntp_adjtime() works in a way similar to the original adjtime() system call, but it runs independently of adjtime(), which keeps working in its traditional fashion. 🔥 When NTP is in use, the design intent is that settimeofday() or adjtime() should only be used for system time adjustments greater than +-128 ms. The dynamic range of the new model is much larger, though, at +-512 ms. 🌸 Internet experience has shown that changing the system time in increments greater than +-128 ms is extremely rare. When it does happen, it's usually because of a hardware or software malfunction, or a system reboot. 😭🫶
Okay so the easiest way to set the time is the settimeofday() system call 💅 but ngl it can, under some conditions, make the clock jump backward 😬. If that's a no-go for you, adjtime() can slew the clock to the new value instead, so it never runs backward and doesn't mess with the frequency discipline process 🦋. Once the system clock is within +-128 ms, the ntp_adjtime() system call takes over for periodic updates, including the time offset, maximum error, estimated error and PLL time constant ✨. With NTP the update interval depends on the measured dispersion and time constant, but the whole scheme is super forgiving. Moderate loss of updates or wobbly update intervals are not serious 🫶.
2.2 Daemon and Application Interface
Unix apps can read the system clock with the gettimeofday() system call, which only gives you the system time and timezone data 🕰️. Some applications really need to know the maximum error of the reported time from all causes: clock reading errors, oscillator frequency errors and accumulated latencies on the path to a primary synchronization source 🤓. Here's the plot twist though. In the new model the PLL adjusts the system clock to compensate for its intrinsic frequency error, so the time errors you'd expect in normal operation are usually much less than the maximum error 🔥.
The programming interface gets a new system call, ntp_gettime(), which returns the system time plus the maximum error and estimated error 💖. It's built for apps that need that info, like distributed file systems, multimedia teleconferencing and other real-time applications 🎀. The interface also includes the ntp_adjtime() system call mentioned earlier, which can read and write kernel variables for time and frequency adjustment, PLL time constant, leap-second warning and related data 👑.
Also, the kernel grows the maximum error by an amount equal to the oscillator frequency tolerance times the elapsed time since the last update 📈. The default engineering parameters are optimized for update intervals on the order of 64 s 🍓. For other intervals you can adjust the PLL time constant to optimize the dynamic response over 16-1024 s, and normally NTP does this automatically for you 🧚. If updates get suspended, the PLL coasts at the frequency it last determined. That usually means errors creep up to only a few tens of milliseconds over a day, using room-temperature quartz oscillators on typical modern workstations 🌸.
While any synchronization daemon could in principle be modified to use the new system calls, the most likely users are the NTP Version 3 daemon xntpd 💜. The xntpd code figures out whether the new system calls are implemented and reconfigures itself automatically as needed 🦋. When they are, the daemon reads the frequency offset from a file and passes it along with the initial time constant via ntp_adjtime() 💅. In later calls to ntp_adjtime(), only the time offset and time constant are affected. The daemon reads the frequency from the kernel using ntp_adjtime() about once an hour and writes it to a system file. After a reboot and daemon restart, that info is recovered, so the sometimes extensive training period to learn each system's frequency can be skipped 🙈✨.
2.3. Precision Clocks for DECstation 5000/240 and 3000 AXP Alpha
The stock microtime() routine in the Ultrix kernel returns system time only to the precision of the timer interrupt interval, which is in the 1-4 ms range 😬. But on the DECstation 5000/240 (and possibly other machines in that family) there's an undocumented IOASIC hardware register that counts system bus cycles at a rate of 25 MHz 👀. The new microtime() routine for the Ultrix kernel uses that register to interpolate system time between timer interrupts. The result is a precision of 1 us for all time values obtained through gettimeofday() and ntp_gettime() 🔥.
For the Digital Equipment 3000 AXP Alpha, the architecture provides a hardware Process Cycle Counter and a machine instruction rpcc to read it 🧠. This counter runs at the fundamental frequency of the CPU clock, or some submultiple of it, like 133.333 MHz for the 3000/400 🦋. The new microtime() routine for the OSF/1 kernel uses this counter the same way the Ultrix routine does 💜.
In both the Ultrix and OSF/1 kernels, the gettimeofday() and ntp_gettime() system calls use the new microtime() routine. It returns the actual interpolated value but does not change the kernel time variable 🫶. So any routines that access the kernel time variable directly, and don't call gettimeofday(), ntp_gettime() or microtime(), keep behaving exactly as they did before 🌈. The microtime() feature is independent of the other features described here, and it works even if the kernel PLL or new system calls haven't been implemented ✨.
The SunOS kernel already has a system clock with 1-us resolution, so in principle no microtime() routine is necessary 💅. There is an existing kernel routine, uniqtime(), that implements this function, but it's coded in C and is rather slow at 42-85 us per call 😭. A replacement microtime() routine coded in assembler language is available in the NTP Version 3 distribution, and it's much faster at about 3 us per call 🏎️💖.
2.4. External Time and Frequency Discipline
The overall accuracy of a time synchronization subnet with respect to Coordinated Universal Time (UTC) depends on two things: the accuracy and stability of the primary synchronization source (usually a radio or satellite receiver) and the system clock oscillator of the primary server 📡. As discussed in [5], the traditional interface using an RS232 protocol and serial port blocks the full accuracy of the radio clock 🙅♀️. The poor stability of typical CPU clock oscillators also limits accuracy, whether or not precision time sources are available 😤. The good news is there are several ways to improve system clock accuracy and stability, up to the limit set only by the accuracy and stability of the synchronization source and the jitter of the operating system 👑.
Many radio clocks produce special signals that external equipment can use to precisely synchronize time and frequency 🎀. Most produce a pulse-per-second (PPS) signal that can be read via a modem-control lead of a serial port. Some produce a special IRIG signal that can be read directly by a bus peripheral, like the KSI/Odetics TPRO IRIG SBus interface, or indirectly via the audio codec of some workstations, as described in [5] 🧚. In the NTP Version 3 distribution, the PPS signal can augment the less precise ASCII serial timecode to improve accuracy to the order of microseconds 🔥. Support is also included in the distribution for the TPRO interface and the audio codec. The latter, however, requires a modified kernel audio driver contained in the bsd_audio.tar.Z distribution, which lives in the same host and directory as the NTP Version 3 distribution mentioned earlier 🌸.
2.4.1. PPS Signal
The NTP Version 3 distribution includes a special ppsclock module for the SunOS 4.1.x kernel. It captures the PPS signal presented via a modem-control lead of a serial port 🔌. Normally, ppsclock produces a timestamp at each transition of the PPS signal and hands it to the synchronization daemon, which integrates it with the serial ASCII timecode also produced by the radio clock 💜. With the conventional PLL implementation (in either the daemon or the kernel, as described above), this scheme's accuracy is limited by the intrinsic stability of the CPU clock oscillator, to a millisecond or two, depending on environmental temperature variations 🥶.
The ppsclock module has been modified to also call a new kernel routine, hardpps(), once each second ⏰. The kernel routine compares the timestamp with a sample of the CPU clock oscillator to develop a frequency offset estimate. That offset is used to discipline the oscillator frequency, nominally to within a few parts in 10^8, which is about two orders of magnitude better than the undisciplined oscillator 🤯✨. This new feature is conditionally compiled into the code described below only if the PPS_SYNC option is used in the kernel configuration file 🙈.
When using the PPS signal to adjust the time, there's a problem with the SunOS implementation that's very delicate to fix 😵💫. The Sun serial port interrupt routine runs at interrupt priority level 12, while the timer interrupt routine runs at priority 10. So the PPS signal interrupt can occur during the timer interrupt routine. When that happens, a tick increment can be missed and the returned time comes out early by one tick. If the CPU clock oscillator is within a few ppm of the PPS oscillator, this condition can persist for two or more successive PPS interrupts 😬. A useful workaround has been to use a median filter on the PPS sample offsets. In this filter, the sample offsets in a window of 20 samples are sorted by offset, the six highest and six lowest outlyers are discarded, and the average of the eight samples remaining becomes the filter's output 🧮💖.
The problem is not nearly as serious when the PPS signal is used to discipline the frequency of the CPU clock oscillator. In that case the quantity of interest is only the contents of the microseconds counter, which doesn't depend on the kernel time variable 🦋✨.
2.4.2. External Clocks
It's possible to replace the system clock function with an external bus peripheral 🤩. The TPRO device mentioned earlier can provide IRIG-synchronized time with a precision of 1 us. A driver for this device, tprotime.c, and header file, tpro.h, are included in the kernel.tar.Z distribution mentioned earlier 💅. With this device, the system clock is read directly from the interface. However, the device doesn't record the year, so special provisions have to be made to get the year from the kernel time variable and initialize the driver accordingly 🗓️. This feature is conditionally compiled into the code described below only if the EXT_CLOCK option is used in the kernel configuration file 🎀.
While the system clock function is provided directly by the microtime() routine in the driver, the kernel time variable must be disciplined as well, since not all system timing functions use the microtime() routine 🧠. This is done by measuring the difference between the microtime() clock and the kernel time variable, then using that difference to adjust the kernel PLL as if the adjustment came from an external peer and NTP 🌈👑.
A good deal of error checking happens in the TPRO driver 🛡️, because the system clock is vulnerable to a misbehaving radio clock, IRIG signal source, interface cables, and the TPRO device itself 😭. Sadly, there's no easy way to use all the diversity and redundancy the NTP synchronization daemon can do 💅. To avoid disruptions when the TPRO time is way off from the kernel time variable, the kernel time variable gets used instead whenever the difference between the two goes over 1000 s ✨. Presumably at that point an operator needs to step in 🙈.
2.4.3. External Oscillators
Even without a PPS or IRIG source, you can still make the system clock more stable with a specialized bus peripheral 🦋. To explore that idea, someone built a special SBus peripheral called HIGHBALL 🎀. It has a pair of 32-bit hardware counters in Unix timeval format, plus a precision, oven-controlled quartz oscillator with a stability of a few parts in 10^9 🔥. The driver for this device, hightime.c, and the header file high.h are included in the kernel.tar.Z distribution mentioned previously 🧚. This feature only gets compiled in the code described below when both the EXT_CLOCK and HIGHBALL options are used in the kernel configuration file ⚙️.
Unlike the external clock case, where the system clock function comes straight from the microtime() routine in the driver, the HIGHBALL counter offsets with respect to UTC have to be worked out first 💖. This happens using the ordinary kernel PLL, but it controls the counter offsets directly instead of the kernel time variable 💜. At first this might seem like it defeats the whole point 🙃, since the jitter and wander of the synchronization source will affect the counter offsets and therefore the accuracy of the time. Honestly though, the PLL reduces the jitter a lot, and the wander is small, especially when you're using a radio clock or another primary server disciplined in the same way 🌈. In practice the scheme reduces the incidental wander to a few parts in 10^8, which is about the same as using the PPS signal 👑.
Just like the previous case, the kernel time variable has to be disciplined too, since not every system timing function uses the microtime() routine 🌸. The catch is that the kernel PLL can't be used for this, because it's already busy providing offsets for the HIGHBALL counters 😤. So instead, a special correction gets calculated from the difference between the microtime() clock and the kernel time variable, and it's applied to the kernel time variable at the next timer interrupt ⏰. This roundabout approach is needed so the adjustment doesn't make the kernel time variable jump backwards, which could lose or duplicate a timer event 🫶.
2.5 Other Features
It's a design feature of the NTP architecture that the system clocks in a synchronization subnet read the same, or nearly the same, values before, during, and after a leap-second event, as declared by national standards bodies ⭐. The new model implements the leap event on command, through an ntp_adjtime() argument 🍓. The intricate and sometimes arcane details of the model and implementation are discussed in [3] and [5] 📚. More details are in the technical summary later in this memo 💅.
3. Technical Summary
To really get how the model works, there's a standalone simulator, kern.c, plus the header file timex.h, both included in the kernel.tar.Z distribution mentioned previously 🧪. There's also a complete C program, kern_ntptime.c, that implements the ntp_gettime() and ntp_adjtime() functions, but the vendor-specific argument-passing code has been deleted 🗑️. The distribution is pretty chunky because of all the comments and ornamentation, so listing these programs in this memo just isn't practical 🙈. Anyway, implementors can totally snip parts of the simulator for new kernel designs, but the formatting conventions would make that tricky if it were printed here 💀.
The kern.c program is an implementation of an adaptive-parameter, first-order, type-II phase-lock loop 🌀. The system clock is built from a set of variables and algorithms defined in the simulator, and it's driven by explicit offsets generated by a driver program that's also included in the program 🎀. The algorithms include code fragments that are almost identical to the ones in the machine-specific kernel implementations and work the same way, but the operations can be understood on their own, separately from any licensed source code those fragments might be built into 🧠. The code fragments themselves aren't derived from any licensed code 🫶. The discussion below assumes you have the simulator code on hand to look at 👀.
3.1. PLL Simulation
The simulator follows the analytical model described in [3] 📐. The main() program acts as a driver for the fragments hardupdate(), hardclock(), second_overflow(), hardpps(), and microtime(), although not every function inside those fragments is simulated 🧩. It simulates the PLL at each timer interrupt and prints a summary of the critical program variables at each time update 🖨️.
The kernel configuration file has three defined options specific to each implementation 🔧. The PPS_SYNC option supports a pulse-per-second (PPS) signal, which is used to discipline the frequency of the CPU clock oscillator ⏱️. The EXT_CLOCK option supports an external kernel-readable clock, such as the KSI/Odetics TPRO IRIG interface or the HIGHBALL precision oscillator, both for the SBus 🔌. The TPRO option supports the former, and the HIGHBALL option supports the latter 💁. External clocks are implemented as the microtime() clock driver, and the kernel configuration file picks the specific source code 📝.
3.1.1. The hardupdate() Fragment
The hardupdate() fragment is called by ntp_adjtime() each time an update is computed, to adjust the system clock's phase and frequency 🔄. Heads up that the time constant is in powers of two, so multiplies can be done with simple shifts 💫. The phase variable is computed as the offset divided by the time constant 🧮. Then the time since the last update is computed and clamped to a maximum for robustness, and set to zero if it's initializing ⏳. The offset is multiplied (sorry about the ugly multiply 🙈) by that result, divided by the square of the time constant, and then added to the frequency variable 📈. Also, all shifts are assumed to be positive, and shifting a signed quantity to the right takes a little dance 💃.
With the defines given, the maximum time offset is set by the size in bits of the long type (32 or 64) minus the SHIFT_UPDATE scale factor (12), or at least 20 bits (signed) 📏. The scale factor is picked so there's no loss of significance in later steps, which may involve a right shift of up to SHIFT_UPDATE bits 🧁. This gives a time adjustment range over +-512 ms ⏰. Since time_constant must be greater than or equal to zero, the maximum frequency offset is set by the SHIFT_USEC scale factor (16), or at least 16 bits (signed) 🌊. That gives a frequency adjustment range over +-31,500 ppm 🚀.
In the addition step, the value of offset * mtemp is not greater than MAXPHASE * MAXSEC = 31 bits (signed), which won't overflow a long add on a 32-bit machine ✅. There could be some loss of precision from the right shift of up to 12 bits, since time_constant is bounded at 6 🤏. The net worst-case frequency resolution ends up around .063 ppm, which isn't significant for most quartz oscillators 💎. The worst case could only happen if the NTP peer misbehaves according to the protocol specification 😬.
The time_offset value is clamped on entry 🔒. The time_phase variable is an accumulator, so it's clamped to the tolerance on every call 🧯. This helps damp transients before the oscillator frequency has been determined, and it also keeps the correctness assertions happy if the time synchronization protocol or implementation misbehaves 🛟.
3.1.2. The hardclock() Fragment
The hardclock() fragment is inserted in the hardware timer interrupt routine, at the point where the system clock is supposed to be incremented ⏲️. Before this fragment runs, the time_update variable has been initialized to the value computed by the adjtime() system call in the stock Unix kernel, normally plus or minus the tickadj value, which is usually on the order of 5 us 🎯. The time_phase variable represents the instantaneous phase of the system clock, and it's advanced by time_adj, which gets calculated in the second_overflow() fragment described below 🌟. If the value of time_phase goes over 1 us in scaled units, time_update is increased by the signed excess, and time_phase keeps the residue 🧃.
Except for an external oscillator such as the HIGHBALL interface, the hardclock() fragment advances the system clock by the value of tick plus time_update ⏩. With an external oscillator, though, the system clock comes directly from the interface, and time_update is used to discipline that interface instead 🛠️. The system clock still has to be disciplined as explained previously, so the value of clock_cpu computed by the second_overflow() fragment gets used instead 🫧.
3.1.3. The second_overflow() Fragment
The second_overflow() fragment goes in at the point where the microseconds field of the system time variable gets checked for overflow ⌛. When an overflow happens, the maximum error time_maxerror is increased by time_tolerance, to reflect the maximum time offset caused by oscillator frequency error 📉. Then the increment time_adj, which advances the kernel time variable, is calculated from the (scaled) time_offset and time_freq variables that were updated at the last call to the hardclock() fragment 🎶.
The phase adjustment is a signed fraction of the time_offset still remaining 🍰. That fraction gets added to time_adj and then subtracted from time_offset 💫. This technique converges quickly when offsets are high, and it still gives good resolution when offsets are low 🏃♀️. The frequency adjustment is the sum of the (scaled) time_freq variable, an adjustment needed when the tick interval doesn't evenly divide one second (fixtick), plus the PPS frequency adjustment pps_ybar (if it's configured) 🌸.
So okay, this part is where the PPS magic happens, the leap-second drama starts, and the clock status state machine finally shows up 🕰️✨
Approximating multiply and divide with shifts is totally fine 💅 except when you need an exact answer. That happens when the PPS signal is disciplining the CPU clock oscillator frequency, which is coming up next. As long as the actual oscillator frequency is a power of two in seconds, no correction is needed 🙌 But in the SunOS kernel the clock frequency is 100 Hz, and that gives an error factor of 0.78 🥲 So the code bumps time_adj up by a factor of 1.25, and the overall error ends up under three percent. Iconic save 🫶
When the day rolls over, the leap-second state machine (more on that below 👀) decides whether a second gets inserted or deleted in the timescale. And microtime() makes sure the reported time is always monotonically increasing, so time never goes backwards on us 🦋
3.1.4. The hardpps() Fragment
Okay, the hardpps() fragment only runs if the PPS_SYNC option is set in the kernel configuration file 🎀 It gets called from the serial port driver (or an equivalent interface) right at the on-time transition of the PPS signal. It works as a first-order, type-I frequency-lock loop (FLL), and it's steering based on the difference between the frequency represented by the pps_ybar variable and the frequency of the hardware clock oscillator. Math with feelings 🧚
Now, to avoid calling microtime() more than once for each PPS transition, the calling program has to grab the system time and the hardware counter contents right at the on-time transition. Then it passes a pointer to the timestamp (Unix timeval) and the counter contents as arguments to the hardpps() call 💜 The hardware counter contents can be figured out by saving the microseconds field of the system time, calling microtime(), and subtracting the saved value. If a counter overflow happened during that process, the resulting microseconds value comes out negative, and in that case the caller adds 1000000 to normalize the microseconds field 🙈
The frequency of the hardware oscillator can be found from the difference in hardware counter readings at the start and end of the calibration interval, divided by the length of the interval. The catch is that the oscillator frequency tolerance, as much as 100 ppm, can make that difference go over the tick value, and that creates an ambiguity 😭 To dodge it, the hardware counter value at the start of the interval gets increased by the current pps_ybar value once each second, but computed modulo the tick value. At the end of the interval, the difference between this value and the value computed from the hardware counter is used as a control signal sample for the FLL 🔥
Control signal samples that exceed the frequency tolerance get tossed, and so do samples that come from excessive interval duration jitter 🗑️ The surviving samples then go through a three-stage median filter. The signal that drives the FLL comes from the median sample, while the average of the differences between the other two samples is used as a measure of dispersion. If the dispersion is below the threshold pps_dispmax, the median is used to correct the pps_ybar value, with a weight expressed as a shift PPS_AVG (2) 👑 On top of the averaging, pps_disp goes up by the amount pps_dispinc once each second. So if the dispersion gets super high, or the PPS signal just dies for some reason, pps_disp will eventually blow past pps_dispmax and raise an alarm 🚨 Very dramatic, very necessary.
At first we don't know an approximate value for pps_ybar, so the calibration interval has to stay small to avoid overflowing the tick 🌸 At the end of each calibration interval, the time difference gets measured. If it's greater than a fraction tick/4, the interval gets cut in half. If it stays under that fraction for four successive calibration intervals, the interval gets doubled. This design adapts on its own to the nominal jitter in the PPS signal, and to the value of tick too, which is honestly so smart 🧠✨ The length of the calibration interval is set by the pps_shift variable, as a shift in powers of two. The minimum value, PPS_SHIFT (2), is picked so that with the highest CPU oscillator frequency of 1024 Hz and frequency tolerance of 100 ppm, the tick won't overflow. The maximum value, PPS_SHIFTMAX (8), is picked so that the maximum averaging time is about 1000 s, based on measurements of Allan variance [5] 📏
Now what happens if the PPS signal fails? Totally fine, bestie 🫶 The current frequency estimate pps_ybar keeps getting used, so the nominal frequency stays correct, subject only to the instability of the undisciplined oscillator. Saving and restoring the frequency estimate goes like this: when the frequency is set from a file, the time_freq value becomes the file value minus the pps_ybar value. When the frequency is retrieved, the two values get added back together before saving to the file 🍓 That makes it a seamless interface, even if the PPS signal fails or the kernel configuration changes. Also, the frequency discipline is active whether or not the synchronization daemon is running. And since every Unix system takes some time after reboot to build a running system, the discipline process has usually settled down by then, with the initial transients from frequency discipline already damped out 😌
3.1.4. External Clock Interface
The external clock driver interface is built from two routines. First there's microtime(), which returns the current clock time. Second there's clock_set(), which furnishes the apparent system time derived from the kernel time variable 🕰️ clock_set() only gets called when the clock is set using the settimeofday() system call, but it can also be called from inside the driver, for example when the year rolls over 🎆
In the stock SunOS kernel and the modified Ultrix and OSF/1 kernels, microtime() returns the kernel time variable plus an interpolation between timer interrupts, based on the contents of a hardware counter. With an external clock, like the one described above, the system clock gets read directly from the hardware clock registers 🔌 Examples of external clock drivers live in the tprotime.c and hightime.c routines included in the kernel.tar.Z distribution 📦
The external clock routines return a status code that says whether the clock is operating correctly, and what the problem is if it isn't. The ntp_gettime() system call interprets that return code, and it moves the status state machine to the TIME_ERR state if an error code comes back. This is the only error checking implemented for the external clock in the present version of the code 🥺
The simulator was used to check the PLL operation across the design envelope of +-512 ms in time error and +-100 ppm in frequency error. That confirms no overflows happen, and that the loop initially converges in about 15 minutes for timer interrupt rates from 50 Hz to 1024 Hz 🎯 The loop has a normal overshoot of a few percent, and a final convergence time of several hours, depending on the initial time and frequency error. Patience is a virtue, even for clocks 🐢💖
3.2. Leap Seconds
It doesn't seem generally useful to expose extra kernel and synchronization protocol details to the user application interface, stuff like stratum, reference identifier, reference timestamp, and so forth 🙅♀️ In theory an application could independently evaluate the quality of time and project how long that time might stay "valid." But doing that properly would basically duplicate the synchronization protocol, and it would need knowledge of a bunch of mundane platform details, like subnet configuration, reachability status, and related variables 😵💫 For the curious, the ntp_adjtime() system call can reveal some of these mysteries 🔍
That said, the user application may need to know whether a leap second is scheduled, because it might affect interval calculations that span the event ⏳ A leap-warning condition gets determined by the synchronization protocol (if remotely synchronized), by the timecode receiver (if one is available), or by the operator (if awake 😴). The synchronization daemon sets this condition on the day the leap second is due (30 June or 31 December, as announced). It does this by specifying a clock status in an ntp_adjtime() system call, either TIME_DEL if a second is to be deleted, or TIME_INS if a second is to be inserted 🎀 Now here's the tea ☕: on all occasions since the leap-second scheme began, there has never been a deletion, and there's not likely to be one in future 🙃
If the value is TIME_DEL, the kernel adds one second to the system time immediately following second 23:59:58, and resets the clock status to TIME_OK. If the value is TIME_INS, the kernel subtracts one second from the system time immediately following second 23:59:59, and resets the clock status to TIME_OOP. In effect, that makes system time repeat second 59 🔁 Right after the repeated second, the kernel resets the clock status to TIME_OK again ✨
Depending on the system call implementation, the reported time during a leap second may repeat (with the TIME_OOP return code set to advertise that fact). Or it may be monotonically adjusted until system time "catches up" to reported time. With the second option, the reported time is correct before and shortly after the leap second (depending on the number of microtime() calls during the leap second), but it freezes or slowly advances during the leap second itself 🫠 Most programs will probably use the ctime() library routine to convert from timeval (seconds, microseconds) format to tm format (seconds, minutes,...). If that routine gets modified to use the ntp_gettime() system call and inspect the return code, it could simply report the leap second as second 60 💅
3.3. Clock Status State Machine
The various options possible with the system clock model in this memorandum need a careful look at the state transitions, status indications, and recovery procedures, in case a crucial signal or interface fails 🚦 So this section presents a prototype state machine designed to support leap second insertion and deletion, and to reveal various kinds of errors in the synchronization process 🔬 The states of this machine are decoded as follows:
TIME_OK If an external clock is present, it is working properly
and the system clock is derived from it. If no external
clock is present, the synchronization daemon is working
properly and the system clock is synchronized to a radio
clock or one or more peers.Okay so the leap-second states are where it gets spicy 🦋
TIME_INS: An insertion of one second into the system clock has been declared right after the last second of the current day, but it hasn't actually happened yet 🌸TIME_DEL: A deletion of the last second of the current day has been declared, but it hasn't been executed yet 💅TIME_OOP: An insertion of one second has been declared after the last second of the current day. The second is in progress but not finished yet. Library conversion routines should interpret this second as23:59:60🕰️✨TIME_BAD: Either (a) the synchronization daemon has said the protocol isn't working properly, (b) every outside source of sync has been lost, or (c) an external clock is present and has just become operational after being non-operational 😭TIME_ERR: An external clock is present, but it's in a non-operational condition 💀
In every state except TIME_ERR, the system clock comes from an external clock if one is present, and from the kernel time variable if not. In TIME_ERR, the external clock is there but not working properly, so the system clock may come from the kernel time variable instead 🫶
Here's the state machine diagram. Not every valid transition is shown, so don't panic if something's missing 🙈
+--------+ +--------+ +--------+ +--------+
| | | | | | | |
|TIME_BAD|---->|TIME_OK |<----|TIME_OOP|<----|TIME_INS|
| | | | | | | |
+--------+ +--------+ +--------+ +--------+
A A
| |
| |
+--------+ +--------+
| | | |
|TIME_ERR| |TIME_DEL|
| | | |
+--------+ +--------+The state machine moves once a second, at the exact moment the microseconds field of the kernel time variable overflows and one second gets added to the seconds field. Here's the tea ☕ That condition only gets checked at each timer interrupt, though, and that check might not line up exactly with the real overflow moment. So you can get some wild anomalies, like a status of leap second in progress (TIME_OOP) when the leap second had actually already ended 😵💫🌈
The kernel runs these transitions automatically:
any state -> TIME_ERR: This happens when an external clock is present and someone tries to read it while it's in a non-operational condition.TIME_INS -> TIME_OOP: This happens right after second 86,400 of the current day, when an insert-second event has been declared.TIME_OOP -> TIME_OK: This happens right after second 86,401 of the current day, which means one second after enteringTIME_OOP🎀TIME_DEL -> TIME_OK: This happens right after second 86,399 of the current day, when a delete-second event has been declared.
These transitions happen through specific ntp_adjtime() system calls:
TIME_OK -> TIME_INS: This happens because of anntp_adjtime()system call that declares an insert-second event.TIME_OK -> TIME_DEL: This happens because of anntp_adjtime()system call that declares a delete-second event.any state -> TIME_BAD: This happens because of anntp_adjtime()system call that declares loss of all sources of synchronization, or in other error cases 🚨
Now for the table, which is honestly the most satisfying part 🍓 It lines up the UTC and NTP times at the start of each second, for three situations. The left column is what happens when no leap event is coming. The middle column is an insertion: the machine is in TIME_INS at the end of UTC second 23:59:59, and NTP time has just hit 400. NTP time gets set back one second to 399, and the machine enters TIME_OOP. At the end of that repeated second, the machine enters TIME_OK and UTC and NTP are back in sync 💖 The right column is a deletion: the machine is in TIME_DEL at the end of UTC second 23:59:58, and NTP time has just reached 399. NTP time gets incremented, the machine enters TIME_OK, and both clocks are back in correspondence ✨
No Leap Leap Insert Leap Delete
UTC NTP UTC NTP UTC NTP
---------------------------------------------
23:59:58|398 23:59:58|398 23:59:58|398
| | |
23:59:59|399 23:59:59|399 00:00:00|400
| | |
00:00:00|400 23:59:60|399 00:00:01|401
| | |
00:00:01|401 00:00:00|400 00:00:02|402
| | |
00:00:02|402 00:00:01|401 00:00:03|403
| | |Finding local midnight is kinda messy 🙃 The kernel code just takes the residue of the time.tv_sec (or time.tv_sec + 1) value mod 86,400, but that needs a messy divide. A better approach is probably to set up an auxiliary counter in the settimeofday() routine, using one ugly divide there, and then bump the counter at the same moment time.tv_sec gets incremented in the timer interrupt routine. That's one for future embellishment 🧚
4. Programming Model and Interfaces ✨
This section covers the programming model for the synchronization daemon and for user application programs. The ideas come from suggestions by Jeff Mogul and Philip Gladstone, plus a similar interface Philip designed 🫶 It's really important that the original Unix adjtime() system call still works the same way, so the modified kernel acts like the unmodified one whenever the new features aren't being used. Even then, the ntp_adjtime() system call can still read and write kernel variables that a synchronization daemon other than NTP might use 🌸
4.1. The ntp_gettime() System Call
The syntax and semantics of the ntp_gettime() call are given in the timex.h header fragment below. This file is identical, except for the SHIFT_HZ define, across the SunOS, Ultrix and OSF/1 kernel distributions. (SHIFT_HZ is the base-2 logarithm of the clock oscillator frequency, and it's specific to each system type 💅) Heads up: the timex.h file pulls in the syscall.h system header file, which has to be modified to define the SYS_ntp_gettime system call for each system type. The kernel distributions include directions for doing that 📖
/*
* This header file defines the Network Time Protocol (NTP)
* interfaces for user and daemon application programs. These are
* implemented using private system calls and data structures and
* require specific kernel support.
*
* NAME
* ntp_gettime - NTP user application interface
*
* SYNOPSIS
* #include <sys/timex.h>
*
* int system call(SYS_ntp_gettime, tptr)
*
* int SYS_ntp_gettime defined in syscall.h header file
* struct ntptimeval *tptr pointer to ntptimeval structure
*
* NTP user interface - used to read kernel clock values
* Note: maximum error = NTP synch distance = dispersion + delay /
* 2
* estimated error = NTP dispersion.
*/
struct ntptimeval {
struct timeval time; /* current time */
long maxerror; /* maximum error (us) */
long esterror; /* estimated error (us) */
};The ntp_gettime() system call hands back three values in the ntptimeval structure: the current time in Unix timeval format, plus the maximum and estimated errors, both in microseconds 🕰️ The 32-bit long type can technically hold error values bigger than an hour, but that doesn't really matter in practice 🙌 The protocol declares an unsynchronized condition way before it gets that big. In the NTP Version 3 specification, if the protocol computes either of these values above 16 seconds, they get clamped to 16 seconds and the system clock is declared unsynchronized 😤
Here's what each ntptimeval member means 👑
struct timeval time; /* current time */This member returns the current system time as a Unix timeval structure. A timeval is two 32-bit words: the first is the number of seconds since 1 January 1970, and the second is the number of microseconds 🦋
long maxerror; /* maximum error (us) */This member returns the time_maxerror kernel variable, in microseconds. Check out the entry for this variable in section 5 for more details 🍓
long esterror; /* estimated error (us) */This member returns the time_esterror kernel variable, in microseconds. Same deal: see the entry for this variable in section 5 for more info 🌈
4.2. The ntp_adjtime() System Call
The syntax and semantics of the ntp_adjtime() call are given in the timex.h header fragment that follows. Just like with the ntp_gettime() system call, the syscall.h system header file has to be modified to define the SYS_ntp_adjtime system call for each system type 💖
Okay bestie, the timex struct is where the magic lives 💅 Here's the full layout, with the NAME/SYNOPSIS header first:
/*
* NAME
* ntp_adjtime - NTP daemon application interface
*
* SYNOPSIS
* #include <sys/timex.h>
*
* int system call(SYS_ntp_adjtime, mode, tptr)
*
* int SYS_ntp_adjtime defined in syscall.h header file
* struct timex *tptr pointer to timex structure
*
* NTP daemon interface - used to discipline kernel clock
* oscillator
*/
struct timex {
int mode; /* mode selector */
long offset; /* time offset (us) */
long frequency; /* frequency offset (scaled ppm) */
long maxerror; /* maximum error (us) */
long esterror; /* estimated error (us) */
int status; /* clock command/status */
long time_constant; /* pll time constant */
long precision; /* clock precision (us) (read only)
*/
long tolerance; /* clock frequency tolerance (scaled
* ppm) (read only) */
/*
* The following read-only structure members are implemented
* only if the PPS signal discipline is configured in the
* kernel.
*/
long ybar; /* frequency estimate (scaled ppm) */
long disp; /* dispersion estimate (scaled ppm)
*/
int shift; /* interval duration (s) (shift) */
long calcnt; /* calibration intervals */
long jitcnt; /* jitter limit exceeded */
long discnt; /* dispersion limit exceeded */
};So ntp_adjtime() is the system call that lets you read and write certain time-related kernel variables 🕰️✨ These are summarised in this section and the ones after it. Writing is superuser-only 🔒, so only root can do it, bestie. To write a variable, you set the mode member with one or more bits, and each bit maps to one of the variables below. The current values for every variable come back no matter what, so a mode of zero just means "show me the values, don't change anything" 🫶
Here's what each structure member does:
int mode; /* mode selector */This is a bit-coded variable that selects one or more structure members, with one bit assigned to each member. If a bit is set, the value of the matching member is copied to the matching kernel variable. If not, that member gets ignored 🙈 The bits are assigned as shown in this fragment of the timex.h header file. Also, the precision and tolerance are decided by the kernel and cannot be changed by ntp_adjtime() 💀
/*
* Mode codes (timex.mode)
*/
#define ADJ_OFFSET 0x0001 /* time offset */
#define ADJ_FREQUENCY 0x0002 /* frequency offset */
#define ADJ_MAXERROR 0x0004 /* maximum time error */
#define ADJ_ESTERROR 0x0008 /* estimated time error */
#define ADJ_STATUS 0x0010 /* clock status */
#define ADJ_TIMECONST 0x0020 /* pll time constant */long offset; /* time offset (us) */If selected, this member replaces the value of the time_offset kernel variable, in microseconds. The absolute value has to be less than MAXPHASE microseconds, which is defined in the timex.h header file. Check the entry for this variable in section 5 for more info 🧚
If it's within range and the PPS signal and/or external oscillator are configured and operating properly, the clock status is automatically set to TIME_OK 🎀
long time_constant; /* pll time constant */If selected, this member replaces the value of the time_constant kernel variable. The value has to be between zero and MAXTC, which is defined in the timex.h header file. See the entry for this variable in section 5 for more details 💜
long frequency; /* frequency offset (scaled ppm) */If selected, this member replaces the value of the time_frequency kernel variable. The value is in ppm, with the integer part in the high-order 16 bits and the fraction in the low-order 16 bits. The absolute value has to be in the range less than MAXFREQ ppm, which is defined in the timex.h header file. See the entry for this variable in section 5 for additional information 🌈
long maxerror; /* maximum error (us) */If selected, this member replaces the value of the time_maxerror kernel variable, in microseconds. See the entry for this variable in section 5 for additional information 🦋
long esterror; /* estimated error (us) */If selected, this member replaces the value of the time_esterror kernel variable, in microseconds. See the entry for this variable in section 5 for additional information 🌸
int status; /* clock command/status */If selected, this member replaces the value of the time_status kernel variable. See the entry for this variable in section 5 for additional information ⭐
To set this variable with ntp_adjtime(), either (a) the current clock status has to be TIME_OK, or (b) the member value is TIME_BAD. Basically, the call can always put the clock into the unsynchronized state or, if the clock is running correctly, put it into any state 👑 Either way, ntp_adjtime() always returns the current state in this member, so the caller can tell whether or not the request worked 🫶
long time_constant; /* pll time constant */If selected, this member replaces the value of the time_constant kernel variable. The value has to be between zero and MAXTC, which is defined in the timex.h header file. See the entry for this variable in section 5 for additional information 💅
long precision; /* clock precision (us) (read only) */This member returns the time_precision kernel variable, in microseconds. The variable can only be written by the kernel. See the entry for this variable in section 5 for more info 🍓
long tolerance; /* clock frequency tolerance (scaled ppm)
*/This member returns the time_tolerance kernel variable, in microseconds. The variable can only be written by the kernel. See the entry for this variable in section 5 for more info 🧚♀️
long ybar; /* frequency estimate (scaled ppm) */This member returns the pps_ybar kernel variable, in microseconds. The variable can only be written by the kernel. See the entry for this variable in section 5 for more info ✨
long disp; /* dispersion estimate (scaled ppm) */This member returns the pps_disp kernel variable, in microseconds. The variable can only be written by the kernel. See the entry for this variable in section 5 for more info 🌷
int shift; /* interval duration (s) (shift) */This member returns the pps_shift kernel variable, in microseconds. The variable can only be written by the kernel. See the entry for this variable in section 5 for more info 🦄
long calcnt; /* calibration intervals */This member returns the pps_calcnt kernel variable, in microseconds. The variable can only be written by the kernel. See the entry for this variable in section 5 for more info 🎀
long jitcnt; /* jitter limit exceeded */This member returns the pps_jittcnt kernel variable, in microseconds. The variable can only be written by the kernel. See the entry for this variable in section 5 for more info 💖
long discnt; /* dispersion limit exceeded */This member returns the pps_discnt kernel variable, in microseconds. The variable can only be written by the kernel. See the entry for this variable in section 5 for more info 🌈
4.3. Command/Status Codes
The kernel routines use the system clock status variable time_status, which records whether the clock is synchronized, waiting for a leap second, and so on 🕐 The value of this variable comes back as the result code from both ntp_gettime() and ntp_adjtime(). It can also be read and written explicitly through ntp_adjtime(), but writing is superuser mode only 🔐 These are the values currently defined in the timex.h header file:
/*
* Clock command/status codes (timex.status)
*/
#define TIME_OK 0 /* clock synchronized */
#define TIME_INS 1 /* insert leap second */
#define TIME_DEL 2 /* delete leap second */
#define TIME_OOP 3 /* leap second in progress */
#define TIME_BAD 4 /* kernel clock not synchronized */
#define TIME_ERR 5 /* external oscillator not
synchronized */There's a detailed description of how these codes work in the leap-second state machine, given later in this memorandum 🧠 For a negative result code, the kernel has intercepted an invalid address or, for the ntp_adjtime() call, a superuser violation 🚨
5. Kernel Variables
This section is the list of kernel variables, plus a detailed rundown of what they do, their initial values, their scaling and their limits 📚
5.1. Interface Variables
The following variables are read and set by the ntp_adjtime() system call. Extra automatic variables are used as temporaries, as described in the code fragments 🧁
int time_status = TIME_BAD;This variable controls the state machine used to insert or delete leap seconds, and shows the status of the timekeeping system, the PPS signal and the external oscillator, if configured ⏱️
long time_offset = 0;This variable is used by the PLL to adjust the system time in small increments. It is scaled by (1 << SHIFT_UPDATE) (12) in microseconds. The maximum value that can be represented is about +-512 ms, and the minimum value or precision is a few parts in 10^10 s 🌊
long time_constant = 0; /* pll time constant */This variable sets the bandwidth, or "stiffness", of the PLL. The value is used as a shift between zero and MAXTC (6), with the effective PLL time constant equal to a multiple of (1 << time_constant) seconds. For a room-temperature quartz oscillator the recommended default value is 2, which corresponds to a PLL time constant of about 900 s and a maximum update interval of about 64 s 🌷 The maximum update interval scales directly with the time constant, so at the maximum time constant of 6, the update interval can be as large as 1024 s ✨
Values of time_constant between zero and 2 can be used if quick convergence is necessary. Values between 2 and 6 can be used to reduce network load, but at a modest cost in accuracy 💅 Values above 6 are only appropriate if an external oscillator is present 🫶
Okay so bestie, the time_tolerance variable is basically the clock's wobble limit 🕰️✨ Here's the declaration:
long time_tolerance = MAXFREQ; /* frequency tolerance (ppm) */This one is giving "how much can my oscillator drift before I get weird" 💅 It's the max frequency error (in ppm) of the CPU clock oscillator. It's basically an architecture thing, but it could change if outside discipline signals are around, for instance. It's always a positive number bigger than zero, measured in parts-per-million (ppm) 🌸
The recommended value of MAXFREQ is 200 ppm, which is the vibe for room-temperature quartz oscillators in normal workstations 🖥️ But it can shift depending on the PPS signal and/or external oscillator conditions. With either a PPS signal or an external oscillator, the recommended value for MAXFREQ drops to 100 ppm 🫶
long time_precision = 1000000 / HZ; /* clock precision (us) */time_precision is the max error when reading the system clock, in microseconds ⏱️ Usually it's based on how many microseconds pass between timer interrupts: 10000 us for the SunOS kernel, 3906 us for the Ultrix kernel, and 976 us for the OSF/1 kernel. BUT if the time can be interpolated between timer interrupts with microsecond resolution (like the unmodified SunOS kernel and the modified Ultrix and OSF/1 kernels), the precision is specified as 1 us 🤯 If a PPS signal or external oscillator is around, the precision can depend on how that signal or oscillator is behaving. The kernel figures this out for the synchronization daemon, but otherwise doesn't touch it 🙈
long time_maxerror = MAXPHASE; /* maximum error */time_maxerror establishes the max error of the indicated time compared to the primary synchronization source, in microseconds 🎀 For NTP, it gets set by an ntp_adjtime() call to the synchronization distance, which is the root dispersion plus one-half the root delay. Every second it grows by a small amount (time_tolerance) to account for clock frequency tolerance. The synchronization daemon and the kernel both compute it, but the kernel doesn't otherwise use it 🧚
long time_esterror = MAXPHASE; /* estimated error */time_esterror is the expected error of the indicated time relative to the primary synchronization source, also in microseconds ✨ For NTP, it's determined as the root dispersion. That's the best guess of the system clock's real error, based on its past behavior, plus observations of multiple clocks in the peer group. The synchronization daemon computes it and it gets returned in system calls, but the kernel doesn't otherwise use it 💜
5.2. Phase-Lock Loop Variables 🔁
These variables set the state of the PLL and the residual time and frequency offset of the system clock. Extra automatic variables get used as temporaries, as described in the code fragments 🧩
long time_phase = 0; /* phase offset (scaled us) */time_phase represents the phase of the kernel time variable at each clock tick 🦋 It's scaled by (1 << SHIFT_SCALE) (23) in microseconds, which gives a max adjustment of about +-256 us/tick and a resolution less than one part in 10^12. Iconic precision, no cap 👑
long time_offset = 0; /* time offset (scaled us) */time_offset is the time offset of the CPU clock oscillator 🕒 It gets recalculated every time a system clock update comes in through the hardupdate() routine, and again every second in the seconds_overflow routine. It's scaled by (1 << SHIFT_UPDATE) (12) in microseconds, so the max adjustment is about +-512 ms, with a resolution of a few parts in 10^10 s 🌈
long time_freq = 0; /* frequency offset (scaled ppm) */time_freq is the frequency offset of the CPU clock oscillator 💖 It gets recalculated with each system clock update through hardupdate(). It can also be set via ntp_adjtime() from a value stored in a file when the synchronization daemon first starts up. The daemon can retrieve it via ntp_adjtime() and write it to the file about once per hour 🗓️ It's scaled by (1 << SHIFT_KF) (16) ppm, so its max value is way bigger than the +-256 ppm limit set by other constraints. Its precision (frequency resolution) is parts in 10^11, which is good enough for everything except the very best external oscillators 🔥
time_adj = 0; /* tick adjust (scaled 1 / HZ) */time_adj is the adjustment added to the value of tick at each timer interrupt 🍓 It's computed once every second from time_offset, time_freq and, if the PPS signal is present, the ps_ybar variable.
long time_reftime = 0; /* time at last adjustment (s) */time_reftime is the seconds portion of the system time at the last update received by hardupdate() 🫧 It's used to compute time_freq as the time since the last update goes up.
int fixtick = 1000000 % HZ; /* amortization factor */fixtick is the one with the plot twist 🎭 In the Ultrix and OSF/1 kernels, the interval between timer interrupts doesn't evenly divide the number of microseconds in a second. So for the clock to run at a precise rate, an amortization factor has to go into the local timescale. In the original Unix code, fixtick got amortized once each second, which added another source of jitter 😭 In the new model, it's amortized at each tick of the system clock instead, which cuts the jitter by the reciprocal of the clock oscillator frequency. This isn't a brand new kernel variable, just a new use of an existing one 🧠
5.3. Pulse-per-second (PPS) Frequency-Lock Loop Variables 📡
These variables only get used if a pulse-per-second (PPS) signal is available and hooked up via a modem-control lead, like the one produced by the optional ppsclock feature in the serial port driver 🔌 They set the design parameters of the PPS frequency-lock loop, which disciplines the CPU clock oscillator to an external PPS signal. Extra automatic variables get used as temporaries here too, as described in the code fragments 🧩
long pps_usec; /* microseconds at last pps */pps_usec gets latched from a high resolution counter or external oscillator at each PPS interrupt 🎯 When figuring out this value, only the hardware counter contents are used, not the contents plus the kernel time variable as returned by the microtime() routine. Main character energy 💅
long pps_ybar = 0; /* pps frequency offset estimate */pps_ybar is the average CPU clock oscillator frequency offset relative to the PPS disciplining signal 🌸 It's scaled in the same units as time_freq.
pps_disp = MAXFREQ; /* dispersion estimate (scaled ppm) */pps_disp represents the average sample dispersion measured over the last three samples 📊 It's scaled in the same units as time_freq.
pps_dispmax = MAXFREQ / 2; /* dispersion threshold */pps_dispmax is used as a dispersion threshold 🚧 If pps_disp is less than this threshold, the median sample gets used to update the pps_ybar estimate. If not, the sample gets tossed 🗑️
pps_dispinc = MAXFREQ >> (PPS_SHIFT + 4); /* pps dispersion
increment/sec */pps_dispinc is the increment added to pps_disp once each second ⏳ It's calculated so that if no PPS samples have shown up for several calibration intervals, pps_disp will go over the pps_dispmax threshold and raise an alarm 🚨
int pps_mf[] = {0, 0, 0}; /* pps median filter */The pps-mf[] array works as a median filter to detect and discard jitter in the PPS signal 🫶
int pps_count = 0; /* pps calibrate interval counter */pps_count measures the length of the calibration interval used to calculate the frequency 🔢 It normally counts from zero up to the value 1 << pps_shift.
pps_shift = PPS_SHIFT; /* interval duration (s) (shift) */pps_shift determines the duration of the calibration interval, which is 1 << pps_shift s ⏲️
pps_intcnt = 0; /* intervals at current duration */pps_intcnt counts the calibration intervals at the current interval duration 🔄 It gets reset to zero after four intervals and whenever the interval duration changes.
long pps_calcnt = 0; /* calibration intervals */pps_calcnt counts the total number of calibration intervals 🧮
long pps_jitcnt = 0; /* jitter limit exceeded */pps_jitcnt counts the resets caused by excessive jitter or frequency offset 🙃 These are usually from too much noise in the PPS signal or interface.
long pps_discnt = 0; /* dispersion limit exceeded */pps_discnt counts the calibration intervals where the dispersion is above the pps_dispmax limit 🌪️ These resets are usually caused by excessive frequency wander in the PPS signal source.
5.4. External Oscillator Variables 🛰️
These variables only get used if an external oscillator (HIGHBALL or TPRO) is present. Extra automatic variables get used as temporaries here as well, as described in the code fragments ✨
int clock_count = 0; /* CPU clock counter */clock_count counts the seconds between adjustments to the kernel time variable, which is how it gets disciplined to the external clock 🕰️
struct timeval clock_offset; /* HIGHBALL clock offset */clock_offset defines the offset between system time and the HIGHBALL counters 🎀
long clock_cpu = 0; /* CPU clock adjust */clock_cpu holds the offset between the system clock and the HIGHBALL clock, which is used for disciplining the kernel time variable 💜
6. Architecture Constants ✨
okay so this section is the cheat sheet for the PLL 🦋 these are the architecture constants that set how fast the loop responds and how stable it stays, plus the hard max bounds that keep the protocol's correctness promises. more definitions live in the timex.h header file too, bestie 🎀
6.1. Phase-lock loop (PLL) definitions 🔥
okay the PLL's performance envelope is defined right here 💅 we get MAXPHASE (max phase error), MAXFREQ (max frequency error), MINSEC (min time between updates) and MAXSEC (max time between updates). the whole point is to keep the PLL inside these limits so the sync protocol's correctness assertions stay valid ✨ if something goes past a bound, it gets clamped to the bound and things keep running normally 🫶 this only really happens if something has failed or is out of tolerance, otherwise the PLL is totally stable and chill 🌈
MAXPHASE MUST be set greater than or equal to CLOCK.MAX (128 ms), as defined in the NTP specification 😭 CLOCK.MAX is the max time offset allowed before the system time gets reset outright instead of nudged incrementally. MAXPHASE is only there to stop overflow errors from defective programming 🙈
MAXFREQ is basically the CPU oscillator's manufacturing frequency tolerance plus the max slew rate the protocol allows 🍓 it SHOULD be at least the oscillator's intrinsic frequency tolerance plus 100 ppm for vernier frequency adjustments 💜 if the kernel frequency discipline code is installed (PPS_SYNC), the CPU oscillator gets disciplined to an external source, which presumably has negligible frequency error 👑
#define MAXPHASE 512000 /* max phase error (us) */
#ifdef PPS_SYNC
#define MAXFREQ 100 /* max frequency error (ppm) */
#else
#define MAXFREQ 200 /* max frequency error (ppm) */
#endif /* PPS_SYNC */
#define MINSEC 16 /* min interval between updates (s)
*/
#define MAXSEC 1200 /* max interval between updates (s)
*/6.2. Pulse-per-second (PPS) Frequency-lock Loop (FLL) Definitions ⭐
these defines only matter if a pulse-per-second (PPS) signal is available and hooked up via a modem-control lead 🔌 like from the optional ppsclock feature in the serial port driver. they set up the design params of the frequency-lock loop (FLL) that disciplines the CPU clock oscillator to the PPS oscillator 🦋
PPS_AVG is the averaging constant used to update the FLL from frequency samples measured each calibration interval 💖 PPS_SHIFT and PPS_SHIFTMAX are the min and max of the calibration interval, written as a power of two 🌸 and PPS_DISPINC is the initial increment to pps_disp each second ✨
#define PPS_AVG 2 /* pps averaging constant (shift) */
#define PPS_SHIFT 2 /* min interval duration (s) (shift)
*/
#define PPS_SHIFTMAX 6 /* max interval duration (s) (shift)
*/
#define PPS_DISPINC 0 /* dispersion increment (us/s) */6.3. External Oscillator Definitions 🧚
these definitions and declarations only come into play if an external oscillator (HIGHBALL or TPRO) is configured on the system 💅
#define CLOCK_INTERVAL 30 /* CPU clock update interval (s) */7. References 📚
[1] Mills, D., "Internet time synchronization: the Network Time Protocol", IEEE Trans. Communications COM-39, 10 (October 1991), 1482- 1493. Also in: Yang, Z., and T.A. Marsland (Eds.). Global States and Time in Distributed Systems, IEEE Press, Los Alamitos, CA, 91-102.
[2] Mills, D., "Network Time Protocol (Version 3) specification, implementation and analysis", RFC 1305, University of Delaware, March 1992, 113 pp.
[3] Mills, D., "Modelling and analysis of computer network clocks", Electrical Engineering Department Report 92-5-2, University of Delaware, May 1992, 29 pp.
[4] Mills, D., "Simple Network Time Protocol (SNTP)", RFC 1361, University of Delaware, August 1992, 10 pp.
[5] Mills, D., "Precision synchronizatin of computer network clocks", Electrical Engineering Department Report 93-11-1, University of Delaware, November 1993, 66 pp.
Security Considerations 🔒
Security issues are not discussed in this memo. 🙈
Author's Address 🎀
David L. Mills Electrical Engineering Department University of Delaware Newark, DE 19716
Phone: (302) 831-8247 EMail: mills@udel.edu
📦 where this page comes from
- ietf: rfc1589 1994-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms