2

Ok, here is the deal, I'm currently using two distributed nagios monitoring servers and over 70 monitored hosts. Each of the hosts is monitored via nrpe addon, each of the hosts have exact same software/hardware configuration. Nrpe is running as a xinetd service on Centos 5.4 Nrpe version is: 2.12 (both server and monitored host) Nagios plugins are: 1.4.14

All nrpe checks are with 5 min intervals

Few of the last entered hosts occasionally show "Service check timed out" with monitored nrpe services.

So what have I done so far:

I've written a few additional wrapper scripts around those services to try and locate the error source, tried debugging on both nagios servers and nrpe servers.

From the debugging the following definitely works:

Network connectivity between hosts Nrpe port is open on both hosts/server No network timeouts whatsoever

Nrpe check is invoked properly, and plugins are working properly I even wrote one logger scripts that logs date and time at the beginning of plugin execution and on the and, also the plugin output when invoked trough nrpe. Execution never exceeds 2 seconds

nrpe.conf is set to wait for command output for 60 secconds, Nrpe plugin on monitoring server is set to wait at least 30 sec for output (even though it would trow entirely diferent error).

Anyhow this lead me to conclusion that something is wrong with nrpe daemon on monitored host. Even if plugin returns the check data nrpe some how don't get it.

I even removed all the shell ulimits for nagios user running the nrpe daemon to eliminate that, but it didn't help.

Anyways any help is very much appreciated.

Improve this question
2
  • Do the nrpe checks ever time out if you run them from the monitoring server's command line (/usr/local/nagios/libexec/check_nrpe -H $HOSTNAME -c $CMD) ? Is there a pattern to which particular nrpe checks time out ? Is there a pattern to which of the two monitoring servers gets the timeouts ?
    – gareth_bowles
    Dec 7, 2009 at 22:38
  • No, no particular pattern. Only pattern is that it happens on newly added hosts to monitor. Only one of the nagios servers is actually doing nrpe checks other one is doing external service checks. Trying the command from shell doesn't return timeouts
    – ModuleC
    Dec 8, 2009 at 16:46

1 Answer 1

Reset to default
0

Ok I finally got it working.

The problem was that both nagios servers are performing service checks and reporting results to master node, and all those check were performed perfectly. Master node had service freshness checking so if the monitoring servers could not complete checks master server would scheduled those checks from itself.

Anyways, new servers were on new ip range and by default nrpe port was closed on master server.

Opening the port solved the problem. Although it's still odd that it returned "Service check timed out" instead of "Socket timeout error".

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .