2

Check_ntp_time has been failing on all my hosts sporadically. I usually receive

CRITICAL - Socket timeout after 10 seconds

And a couple of minutes later check_ntp_time succeeds and shows the correct offset

NTP OK: Offset 0.0001899003983 secs

I've tried raising the check_ntp_time command timeout to 20 seconds but it fails at the same rate. I've tried removing nopeer and noquery from the ntp.conf to no avail (which makes sense because it would fail 100% of the time if that was the issue). The fact that it is failing at random times and succeeding right after is really throwing me off. It's worth noting also that it doesn't fail for all hosts at the same time, it usually fails between 1 to 3 hosts at a time. Any idea what could be causing this?

My check_ntp_time command looks like this:

define command{
    command_name    check_ntp_time
    command_line    $USER1$/check_ntp_time -H pool.ntp.org -t 20 -w 1 -c 3
    }

EDIT:
Metric Min. Max. Average
Check Execution Time: 0.00 sec 20.00 sec 1.153 sec
Check Latency: 0.00 sec 0.00 sec 0.000 sec
Percent State Change: 0.00% 31.84% 0.86%

2 checks per second (0.5 per CPU)

6
  • I would suspect that your NRPE connection is timing out and that NTP is not actually failing. You could validate this by running that check locally in cron and sending the output to an appended log file. Perhaps your nagios server is making too many simultanious checks and too many connections are in TIME_WAIT?
    – Aaron
    Sep 11, 2015 at 17:00
  • Perhaps. The odd thing is that it worked perfectly fine for 6 months and just started recently (I believe after I updated the plugins to the latest version). Is there anything I can modify to go around the issue? Note that I have 266 checks, which is not a huge amount. Maybe 30 of those are check_ntp_time. Sep 11, 2015 at 17:25
  • To get around the issue, you would first have to find out what the issue is. You only have the symptom currently. Perhaps tcpdump between the server and client and look for something preventing communication. Anything else I could add would be purely speculation and feeling around in the dark. Also, netstat -s combined with 'watch -d -n interval' to see if you are getting blocked sockets or tcp timeouts.
    – Aaron
    Sep 11, 2015 at 18:19
  • Sporadic behavior can be caused by resource problems on the Nagios box, as Aaron suggests. Are your trending/graphing nagiosstats? What's your average check latency/execution? How many checks/second?
    – Keith
    Sep 14, 2015 at 15:14
  • I've added the latency and other metrics. Sep 15, 2015 at 16:56

2 Answers 2

1

It's because the check tries to connect over IPv6 a half of timeout specified by '-t' and then it falls back to IPv4. So you can decrease the timeout to 10 seconds and you should get a response in 5 seconds:

[root@server ~]# time /usr/lib64/nagios/plugins/check_ntp_time -q -H time1.google.com -w 1 -c 2 -t 10
NTP OK: Offset 0.0004314184189 secs|offset=0.000431s;1.000000;2.000000;

real    0m5.767s
user    0m0.843s
sys     0m4.908s

Or you can use IPv4 only by '-4', then you will get a response in < 1 sec:

[root@server ~]# time /usr/lib64/nagios/plugins/check_ntp_time -q -H time1.google.com -4 -w 1 -c 2 -t 10
NTP OK: Offset 0.0006598234177 secs|offset=0.000660s;1.000000;2.000000;

real    0m0.401s
user    0m0.003s
sys     0m0.007s
0

This has been resolved by changing the ntp server I had configured to check for the correct time. It's probable that all the servers queried it at the same time and it blocked some requests. I ended up pointing to an internal ntp server that itself is taking its time from the source previously used.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .