Is there a existing mechanism that synchronizes a linux system with NTP while online, and with a predictably drifting RTC while offline?
We operate remote "collectors": embedded Linux systems that collect and timestamp sensor data. We need their clock errors to stay reasonably small, say below 5 seconds. Usually we use NTP to sync their clocks, and that works fine - as long as the system is online.
The problem is that some collectors have very bad uplinks which can go down for hours, days or even weeks. That doesn't stop the local data collection, but without NTP, the Linux system clock drifts badly and quite unpredictably.
OTOH, the hardware's RTC drifts heavily too, but at a constant rate. The RTC drift rate varies from board to board, but is constant per board and can be measured.
I guess what we need is a mechanism that does the following:
- Measure the RTC drift rate of a board before its deployment
- Adjust system time ongoing/regularly via NTP when possible
- Adjust system time regularly from RTC when NTP is unavaiable. Take known RTC drift rate into account.
- Optional: Measure and record the RTC drift rate ongoing while being online (1)
With 'mechanism' i mean some well-maintained, documented piece of software and/or config that can handle the two states "online" vs. "offline", ensure that the system clock is synchronized with the correct time source (ntp vs. rtc), detect change of state, and correct for the RTC drift. It doesn't matter much whether it is implemented as a special ntpd configuration/plugin, as a separate daemon, as a cron job, or else.
I had a look at Chrony, but according to its documentation it tries to predict the drift of the system clock, which in our case drifts far more unpredictably than the RTC. Chrony seems to use the RTC only to keep time across reboots.
(1) Note ntpd activates the kernel's '11-minute mode' (update rtc from system clock every 11 minutes). There seems to be no ways with current kernels and ntpd to prevent the 11-minute mode. Therefore, any rtc drift information gets lost while ntpd is running (thx @billthor).
- We are considering to add an external radio clock for the MSF or DCF77 signal (we are based in Europe) via USB or Serial. But we rather keep the hardware lean.
- Our collectors are located indoors, often in the basement. So adding GPS clocks won't help.
- We use Debian 7. That means hwclock from util-linux-2.20.1, ntpdate-4.2.6p5, ntpd from ntp-4.2.6.p5, chrony-1.24 (potentially 1.30).
- Note that our problem is not that we don't know how to use
date(1), etc. Please see the added section in italics about what i mean with 'mechanism'.
- Added footnote about the '11-minute mode'
- Here is a very interesting discussion about offline-sync and RTC drift