11

I just noticed by pure chance that one of my Cisco 4500 switches has its clock going wrong: it is more than 2 minutes behind in spite of seemingly functional ntp. In my opinion, even a single second should not be considered acceptable for the systems involved. Also, I wouldn't have noticed the difference from diagnostics, had I not compared it to a simple wall-clock.

Some details

Here's ntp information for some of my hosts (10.0.99.1, 10.0.99.2, 10.0.1.119, 10.0.99.241) that are partly referencing one another for fallback, but mainly should all ultimately by syncing with 10.0.0.1, which again pulls the time from outside. So the time discrepancy cannot result from different original time sources. As the observations made me somewhat paranoid, "has correct time" in the following means: show clock (or date) produced an output that matches my wall-clock and my local system clock (which is fine according to http://time.is) with an error certainly below 1 seconds (accuracy of me hitting ENTER while watching my local clock)

10.0.1.119 (Ubuntu) has correct time

$ ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+10.0.99.1       10.0.0.1         3 u  855 1024  377    0.904   -2.658   0.113
*10.0.0.1        130.149.17.8     2 u  266 1024  377    0.253    0.909   0.127

10.0.99.241 (Cisco 2960) has correct time

#sho ntp associations 

  address         ref clock       st   when   poll reach  delay  offset   disp
*~10.0.99.1       10.0.0.1         3     28     64   377  1.462  85.288 19.758
+~10.0.99.2       10.0.1.119       4     29     64   377  1.297  83.515  5.369
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

10.0.99.2 (Cico 4500) has correct time

#sho ntp associations 

  address         ref clock       st   when   poll reach  delay  offset   disp
+~10.0.99.1       10.0.0.1         3      6   1024   111  1.148  -1.618 42.875
*~10.0.1.119      10.0.0.1         3     31   1024   377  0.043   1.687  1.064
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

10.0.99.1 (Cisco 4500) lags behind by about 2 minutes 6 seconds

#sho ntp associations 

  address         ref clock       st   when   poll reach  delay  offset   disp
*~10.0.0.1        130.149.17.8     2    274   1024   377 15.625   3.681 30.403
+~10.0.99.2       10.0.1.119       4    415   1024   376 15.625   0.855 33.276
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

#sho ntp status 
Clock is synchronized, stratum 3, reference is 10.0.0.1      
nominal freq is 250.0000 Hz, actual freq is 249.9988 Hz, precision is 2**6
reference time is DAD8B428.54C6BAEA (20:36:24.331 MESZ Sat May 7 2016)
clock offset is 3.6818 msec, root delay is 32.80 msec
root dispersion is 71.74 msec, peer dispersion is 30.40 msec
loopfilter state is 'CTRL' (Normal Controlled Loop), drift is 0.000004720 s/s
system poll interval is 1024, last update was 683 sec ago.

Questions

  1. How come 10.0.99.1 is so far off?
  2. How come systems that sync to 10.0.99.1 are correct?
  3. How should I learn from the output of sho ntp status on 10.0.99.1 that the clock is actually totally out of sync (compared to all hosts and reference clocks mentioned in sho ntp asso)? For me the output looks totally like a very elaborate "I am totally happy".

EDIT: By popular demand, the output of sho clock detail

10.0.99.1

#sho clock detail 
13:06:38.605 MESZ Tue May 10 2016
Time source is NTP
Summer time starts 02:00:00 MEZ Sun Mar 27 2016
Summer time ends 03:00:00 MESZ Sun Oct 30 2016

10.0.99.2

#sho clock detail 
13:10:54.083 MESZ Tue May 10 2016
Time source is NTP
Summer time starts 02:00:00 MEZ Sun Mar 27 2016
Summer time ends 03:00:00 MESZ Sun Oct 30 2016
  • I can't spot any system in which IP addresses you have configured as ntp servers used by each device. And I spot a loop as well as a couple using each other as ntp servers. I believe in those cases you are supposed to specify them as ntp peers rather than servers. Though I must admit that I don't know what exactly the difference it does whether you specify it as peer or server. Also, I am not convinced it is a good idea to let everything synchronize through a single host (10.0.0.1). But I don't think any of my observations can directly explain the cause of your current problem. – kasperd May 7 '16 at 21:40
  • 2
    One glaring problem with your ntp configuration is that each host is configured with the worst possible number of time sources. "A man with one watch knows what time it is, a man with two watches is never sure..." Any other number is better than two, four is probably the best choice, it gives a cushion if one is unavailable and still leaves three sources. – dfc May 8 '16 at 4:52
  • 4
    Your whole NTP configuration needs to be reconsidered. You need to work with stratum levels. As @kasperd pointed out, you could have a problem with a loop. You should only synchronize to servers with a lower stratum level, and those at the same stratum level could be peered, but not use each other as servers. Peered devices still need one or more servers at a lower stratum level as authoritative source(s), but will try to align themselves to other peers. Don't use busy devices (e.g. core switches) as NTP servers. – Ron Maupin May 10 '16 at 15:03
  • 3
    Something very odd is going on. All the ntp output is reasonably normal and shows good sync. Yet your command to get the time from the device gave a time that's way off. That suggests that for some reason, the device with the time that's off is not setting its system clock from its ntp subsystem. – David Schwartz May 10 '16 at 16:18
  • 1
    It really sounds like you've found a bug, and probably the only way forward is to reboot it and hope it goes away or to contact Cisco. – derobert May 14 '16 at 6:53
2

I am a bit reluctant to post this as an answer because the original cause is still unclear. Nevertheless, the problem seems to be solved - at least for the moment.


Following the comments made by htm11h, I decided to update the firmware. And indeed, now that I am running with a newer firmware, the clock seems to match the correct time.

But does that mean the new firmware was the solution? Unfortunately, no. In my first attempt to load the new firmware, I forgot to change the config register, which was still on its factory default. Therefore, my first reboot ended up in the same original ROM image the router had been running for almost four years (i.e., since its initial power-on). And yet, this was sufficient for the clock to make one huge adjustment and then stay in sync. This suggests that a mere reboot might have helped - temporarily. In turn, this means that the now correct time shown with the newer firmware may still drift away from ntp time over the years to come. It will take a few days until I can safely tell whether or not the clock lost about 5 seconds per day ...

For now, the case is closed.

0

I've done quite a bit of work with the NTP Pool project since the mid-90's and run several NTP Stratum-1 GPS Synced servers here. As others have stated you need more than 2 servers to get time from. I usually use 4 here for the reasons stated by Ron Maupin above. Also as listed you need to look out for loops and setting thing as servers vs. peers.

The time drift could be due to a known bug in IOS that was fixed in this IOS update dealing with the ntp.drift not getting deleted or updated correctly and thus the drift issue. Also 4 YEARS with no reboot or update must have left you in a pretty bad spot security wise as IOS Security updates come out fairly frequently.

Here's an excellent post on setting up NTP on Cisco IOS http://packetlife.net/blog/2011/mar/28/cisco-ios-clocks-and-ntp/

Hope this is helpful. Please ask if you have more questions or issues.

0

Full disclosure: I've only occasionally fiddled with switch configs at all, and I'm not by any means an NTP expert.

That said, I used to see the NTP daemon on RHEL 5.x systems (yes, I'm going back, but you did say your switch had a ~4 year old image...) get stuck in a "happy" state, where it seemed to think it was perfectly synchronized but was clearly not. We would use a ClusterSSH session to run "date" on all of the systems simultaneously, and that would sometimes show as much as 5 minutes of drift between systems. If I recall correctly, we could only seem to fix the problem by restarting the daemon, and ultimately just made cron restart the service every night...

Not by any means an ideal solution, but you might be able to adopt a similar approach with a cron job to connect to the switch and initiate a reboot, or somehow "kick" the NTP daemon on the switch?

Hope this helps!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.