Verify Proper Key setup for the Nagios User:
I use check_by_ssh for all sort of checks and it works fine. First make sure you have ssh keys set up right. Switch to the nagios user with the exact following syntax (You need the env variables I think):
su - nagios
Then ssh to the other box to make sure there is no issue with key based authentication.
The quoting / expansions can be a little hard to figure out sometimes. The key to getting past this is to enable full debugging so you can see what things expand to. To do this set the following three things in your nagios.cfg and then tail -f your debug file:
# DEBUG VERBOSITY
# This option determines how verbose the debug log out will be.
# Values: 0 = Brief output
# 1 = More detailed
# 2 = Very detailed
# DEBUG FILE
# This option determines where Nagios should write debugging information.
You might need to filter through grep because you end up with a fair amount of output.
Reference for Check By SSH:
Also for you reference, and example working check_by_ssh command:
command_line $USER1$/check_by_ssh -t 300 -H $HOSTADDRESS$ -C "$USER1$/checkBandwidth.pl -w $_HOSTPARAM_BAND_W$ -c $_HOSTPARAM_BAND_C$ -s 8 -d"
Edit, Maybe use IPTables to bypass the nesting:
Oh, I missed the whole nesting part...sigh. Either way, checking the keys from both parts as well the debug log stand. However, since with the nested check_by_ssh is running a command you might not get the environment variables you need. You might consider using iptables on the middle host to forward a port to bypass the whole nesting issue.
Maybe specifying the key file in the nested check by help. I think the following option to check_by_ssh might help:
identity of an authorized key [optional]