Ok, here is the deal, I'm currently using two distributed nagios monitoring servers and over 70 monitored hosts. Each of the hosts is monitored via nrpe addon, each of the hosts have exact same software/hardware configuration. Nrpe is running as a xinetd service on Centos 5.4 Nrpe version is: 2.12 (both server and monitored host) Nagios plugins are: 1.4.14
All nrpe checks are with 5 min intervals
Few of the last entered hosts occasionally show "Service check timed out" with monitored nrpe services.
So what have I done so far:
I've written a few additional wrapper scripts around those services to try and locate the error source, tried debugging on both nagios servers and nrpe servers.
From the debugging the following definitely works:
Network connectivity between hosts Nrpe port is open on both hosts/server No network timeouts whatsoever
Nrpe check is invoked properly, and plugins are working properly I even wrote one logger scripts that logs date and time at the beginning of plugin execution and on the and, also the plugin output when invoked trough nrpe. Execution never exceeds 2 seconds
nrpe.conf is set to wait for command output for 60 secconds, Nrpe plugin on monitoring server is set to wait at least 30 sec for output (even though it would trow entirely diferent error).
Anyhow this lead me to conclusion that something is wrong with nrpe daemon on monitored host. Even if plugin returns the check data nrpe some how don't get it.
I even removed all the shell ulimits for nagios user running the nrpe daemon to eliminate that, but it didn't help.
Anyways any help is very much appreciated.