Nest check_by_ssh in Nagios?

Question

I'm trying to monitor a remote cluster of servers with one server that has a public IP. I'm trying to nest check_by_ssh to check on things like load, etc. But no matter what kind of quoting and escaping I do, I can't seem to get it to work.

Here's my latest try:

command_line $USER1$/check_by_ssh -H [gatewayhost] -C '$USER1$/check_by_ssh -H $HOSTADDRESS$ -C "$USER1$/check_load -w $ARG1$ -c $ARG2$"'

But Nagios complains about "could not open pipe". Other combinations of quotes and backslashes results in either the same error or complains about check_by_ssh parameters in either the outer or inner invocation. How do I get this to work?

I could use NRPE I suppose, but I much prefer check_by_ssh.

Answer 1

NRPE is one option that should work. But you could also use NSCA to setup distributed monitoring where you run Nagios, with the NSCA daemon and minus the web interface, on the machine that has a public IP at the remote site. The added bonus with NSCA is that the communication is encrypted at least.

Another option is to setup a VPN tunnel between the sites at which point you can directly monitor the computers in the cluster.

Answer 2

You could set up a simple check script on your public server that takes an ip address and secondary check command. Then nagios could call, for instance: check.sh 192.168.1.12 check_load.

Check.sh would then ssh to the ip address you passed, run the command you passed, and return the result to nagios

Answer 3

Verify Proper Key setup for the Nagios User:
I use check_by_ssh for all sort of checks and it works fine. First make sure you have ssh keys set up right. Switch to the nagios user with the exact following syntax (You need the env variables I think):

su - nagios

Then ssh to the other box to make sure there is no issue with key based authentication.

Quoting Issues:
The quoting / expansions can be a little hard to figure out sometimes. The key to getting past this is to enable full debugging so you can see what things expand to. To do this set the following three things in your nagios.cfg and then tail -f your debug file:

debug_level=-1
# DEBUG VERBOSITY
# This option determines how verbose the debug log out will be.
# Values: 0 = Brief output
#         1 = More detailed
#         2 = Very detailed
debug_verbosity=2
# DEBUG FILE
# This option determines where Nagios should write debugging information.   
debug_file=/usr/local/nagios/var/nagios.debug

You might need to filter through grep because you end up with a fair amount of output.

Reference for Check By SSH:
Also for you reference, and example working check_by_ssh command:

command_line $USER1$/check_by_ssh -t 300 -H $HOSTADDRESS$ -C "$USER1$/checkBandwidth.pl -w $_HOSTPARAM_BAND_W$  -c $_HOSTPARAM_BAND_C$ -s 8 -d"

Edit, Maybe use IPTables to bypass the nesting:
Oh, I missed the whole nesting part...sigh. Either way, checking the keys from both parts as well the debug log stand. However, since with the nested check_by_ssh is running a command you might not get the environment variables you need. You might consider using iptables on the middle host to forward a port to bypass the whole nesting issue.

Maybe specifying the key file in the nested check by help. I think the following option to check_by_ssh might help:

 identity of an authorized key [optional]
 -O, --output=FILE

Answer 4

Aha, I finally figured it out! Thanks to everybody for their suggestions. In the end, I decided instead of trying to use quotes for the inner check_by_ssh command, to instead escape the spaces. Double blackslashes worked, so the final working nagios.cfg line is:

command_line $USER1$/check_by_ssh -H [gatewayhost] -C "$USER1$/check_by_ssh -H $HOSTADDRESS$ -C $USER1$/check_load\\ -w\\ $ARG1$\\ -c\\ $ARG2$"

To ease with trying to figure everything out, I ended up writing out an expansion of the parameters into a Bash script and tweaked it until it finally worked.

Answer 5

Quoting

check_by_ssh -H $HOSTADDRESS$ -C '$USER1$/check_load -w $ARG1$ -c $ARG2$'"