I manually synced the times (and matched hardware times) on two servers A and B.
NTP was running on both machines, so I made sure the /etc/ntp.conf files were identical (the time servers on B were different, so I changed to match server A).
A few weeks later, server B is 1 min 5 sec behind server A!
So this time, I stopped the ntp services on both machines, and manually synced again.
I suppose it's possible I forgot to restart the NTP service on server B, not sure...so I guess I'll start it up again and see if the problem presents itself once more. However, I highly doubt that even if I was using two different time servers, that they would be off by over 1 minute.
Is there any other reason the drift would be so bad?
One last note...when restarting, the output on screen is slightly different...not sure if that matters at all. What is that a function of?
Server A: Starting NTP server: ntpd.
Server B: [ ok ] Starting NTP server: ntpd.
UPDATE: I found another thread that suggests running ntp manually, and with their examples, this is what happens:
:/var# /usr/sbin/ntpdate 192.168.0.30
27 Oct 21:11:34 ntpdate[11553]: no server suitable for synchronization found
root@:/var# /usr/sbin/ntpdate pool.ntp.org
27 Oct 21:11:54 ntpdate[11822]: no server suitable for synchronization found
UPDATE #2: It appears the proper port may be closed:
root@:/var/xxx# nc 127.0.0.1 123 < /dev/null; echo $?
(UNKNOWN) [127.0.0.1] 123 (ntp) : Connection refused
1
I tried to "add" it to the ip tables using this command:
iptables -I INPUT -p udp --dport 123 -j ACCEPT
But it didn't do anything.
UPDATE #3: The last time the NTP drift files were modified was January 2014. Yet it shows the NTP service as running...on all our servers too. Still waiting to find out from our host if UDP 123 is open. Interestingly, the clock correctly changed during daylight savings time...but maybe that is not an NTP thing?