We occasionally have seen time differences on our servers, and confirmed that:
- ntpd crashed without any traceable logs
- ntpq process was dead, but pid existed at /var/run/ntpd.pid
ntpq -p, problem solved
ntpq -p returned
ntpq: read: Connection refused, so I went ahead and
ps aux | grep ntp returned no ntp process, while other working hosts returned something like
/usr/sbin/ntpd -p /var/run/ntpd.pid -u 101:103 -g. It seemed that ntpd actually crashed since no logs seen in /var/log/messages, but it's possible that it happened too long ago and that part in the log was already rotated.
So I went on to
/etc/init.d/ntp restart and was told that the stale pid existed:
Stopping NTP server: ntpdstart-stop-daemon: warning: failed to kill 2124: No such process`. Starting NTP server: ntpd.
but everything got back in place.
We're on Debian 6 Squeeze but the problem has been around since Debian 5 Lenny. We installed ntp using
aptitude install ntp. Servers are on Linode VPS (= Xen virtualization), so we asked them but they said they had no experience like this.
Also, though I don't know if it's just a coincidence or not, it seems that it happens only on application servers (Ruby on Rails) so far.
Thing is, since the pid file remains when ntpd crashes, it's pretty hard to detect the crash and restart with monit or alike. Should I call
/etc/init.d/ntp restart every once in a while by cron?
Any experiences, solutions, thoughts?