15

What is the strategy for detecting time drift in all linux based data centre? This is a more difficult problem than it seems at first.

Time drift can cause serious problems for certain applications and often, even though NTP is installed, it's possible to fail for the following (and many more) reasons:

  • NTP was not correctly set up to automatically restart on reboot.
  • The settings on a server are incorrect so the time server it points to is unreachable or inaccurate.
  • The master time server is unreachable and all servers are syncing with it are now syncing to an unreliable source.

I would like a way to detect if all the individual servers are correct. Bear in mind that the server with the testing script/application may not be right.

Improve this question
2
  • 6
    I think your reasons are bogus, operational NTP is critical for the farm. Ensure what NTP is working properly. Also, deploy a local master clock.
    – user539484
    Sep 2, 2013 at 5:22
  • NTP works out of the box on many Linux distro's now with sensible defaults. It's pretty much a non-issue now.
    – hookenz
    Sep 9, 2013 at 2:14

2 Answers 2

Reset to default
20

This is easy to control. Configuration management is the key...

Ensure that the ntp service is running and configured...

For example, using Monit to make sure ntpd is running and to restart it if it fails is an easy approach... It may make sense to add cron and other essential daemons to that sort of check.

Another option is using a configuration management tool like Puppet to force the same ntpd.conf to your servers and ensure that ntpd is installed, configured and running.

There are enough redundancies in the NTP protocol to deal with the instance of a time server being unreachable. Specify multiple sources.

Improve this answer
4
  • 3
    I have to echo this sentiment. Configuration management and multiple NTP servers is a pretty basic pieces of the infrastructure in anything approaching a "data center". A medium or larger data center should have a hardware clock. GPS time sources can be purchased for a few hundred, and if you've got a soldering iron <$100.
    – Chris S
    Sep 2, 2013 at 4:20
  • this solution + nagios
    – that guy from over there
    Sep 2, 2013 at 6:28
  • 4
    Note that how you check ntpd's status is pretty important. You don't want to just say "ntpd is running. Everything's great!". You must actually verify that your system is synchronizing and has a functional peer, or that the time offset is minimal between the host and the monitoring box. Nagios has check_ntp_peer and check_ntp_time which handle that, if you want to use Nagios.
    – voretaq7
    Sep 3, 2013 at 19:47
  • ntpq to check the status
    – kubanczyk
    Sep 9, 2013 at 5:44
10

There are a variety of check_ntp plugins for nagios out there.

Here's one:

http://nagiosplugins.org/man/check_ntp

Add this check to your nagios host and get alerts if anything goes awry.

Improve this answer
1
  • Yup, just setup your monitoring system to check that the time is valid, or at least the offset relative to the monitoring system is minimal.
    – Zoredache
    Sep 2, 2013 at 2:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .