2

Im an intern sysadmin for a small company. There's no real sysadmin in the place for me to ask when i encounter problems. Thanks for the help

The company uses Nagios to monitor their web server. They use connect_by_ssh to do so with public and private keys. The problem is that sometime it works, sometime it doesnt. Someone can always log in using name and password. its just the keys that dont always work.

Some log for you :

Jan 16 13:23:10 localhost nagios3: SERVICE ALERT:
Server02;SSH;CRITICAL;SOFT;1;Connection timed out

Jan 16 13:24:10 localhost nagios3: SERVICE ALERT:
Server02;SSH;CRITICAL;SOFT;2;Connection timed out

Jan 16 13:24:50 localhost nagios3: SERVICE ALERT:
Server02;SSH;OK;SOFT;3;SSH OK - OpenSSH_5.3 (protocol 2.0)

Jan 16 14:15:10 localhost nagios3: SERVICE ALERT:
Server02;SSH;CRITICAL;SOFT;1;Connection timed out

Jan 16 14:15:50 localhost nagios3: SERVICE ALERT:
Server02;SSH;OK;SOFT;2;SSH OK - OpenSSH_5.3 (protocol 2.0)

Just to be sure, even if the ssh works with user/password

nmap server02.8p-hosting.com

Starting Nmap 5.00 ( http://nmap.org ) at 2014-01-16 14:16 EST
Interesting ports on abc.domain.com (xxx.xxx.xxx.xxx):
Not shown: 971 closed ports
PORT     STATE    SERVICE
22/tcp   open     ssh

Heres how it looks in a regular week :

ssh this week

What could it be?

Log/Debug

ssh -vvv [email protected] OpenSSH_5.5p1 Debian-6+squeeze4, OpenSSL 0.9.8o 01 Jun 2010 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to abc.domain.com [xxx.xxx.xxx.xxx] port 22. debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection timed out ssh: connect to host abc.domain.com port 22: Connection timed out 
11
  • It's hard to say without seeing the command it executes. Is there a reason they opted to use connect_by_ssh over NRPE?
    – CIA
    Jan 16, 2014 at 19:43
  • Most likely they did not know better? Jan 16, 2014 at 19:45
  • I actually prefer check_by_ssh for a lot of things. Has the advantage of being able to execute an event handler on the target system.
    – dmourati
    Jan 16, 2014 at 19:47
  • @CIA i asked my boss who's the one who installed it a while ago. He didn't even know he was using connect_by_ssh...but doesn't want me to change it. When i look at last year data for ssh, its down about 22% of the time. Here's what i got when running it in command line with -vvv option : Jan 16, 2014 at 19:47
  • ssh -vvv [email protected] OpenSSH_5.5p1 Debian-6+squeeze4, OpenSSL 0.9.8o 01 Jun 2010 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to abc.domain.com [xxx.xxx.xxx.xxx] port 22. debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection timed out ssh: connect to host abc.domain.com port 22: Connection timed out Jan 16, 2014 at 19:49

3 Answers 3

0

This looks more like a timeout issue than anything to do with SSH itself.

Take a look at your nagios checks.

You probably want to add a -t option to check_by_ssh:

 -t, --timeout=INTEGER
    Seconds before connection times out (default: 10)

You should probably also check service_check_timeout in your nagios.cfg.

Mine is set to 60s.

http://nagios.sourceforge.net/docs/nagioscore/3/en/configmain.html

1
  • The timeout is 30 sec. with 4 try Jan 16, 2014 at 20:07
0

Unfortunately it could be any number of things, first thing I'd do is turn up the ssh logging on the ssh server to 'DEBUG'.

Also, I'm assuming you mean that they're using check_ssh to monitor the ssh server on the boxes. Inside nagios, there are a couple of things you can do to see what command is being executed exactly. If you have ssh access to the nagios server, you can just login and look at the nagios services.cfg, to find exactly what nagios plugin is being called, with exactly which switches.

Then look at commands.cfg to see what executing. Then, try using that command to test the ssh server manually from the command line.

The other way is using nagios' interface. On the nav bar on the left, at the bottom is a configuration link. Click on it, then using the drop down, go to services, and find exactly what plugin is being called for that service. Next using the dropdown goto command expansion and get the command that way. Then manually check.

Lastly, look to see if SELinux is enabled, if so, the selinux context probably needs changed on the file. If you're using something like puppet or chef, it's possible it's fighting over the file being fixed then broken. Etc.

UPDATE:

I would try adding -E and/or -S to the check_by_ssh command. Sometimes weird ssh output can mess up the connection if it thinks it should be waiting. Also, adding in -v will give you an indication of what's going on.

7
  • check_ssh /usr/lib/nagios/plugins/check_ssh -t 30 '$HOSTADDRESS$' Is the command being used. Since its working atm, i cant give you the output if i put it in command line. Jan 16, 2014 at 20:00
  • What check exactly is failing? For what service? Are all nagios checks failing intermittently, or just a specific service check.
    – rfelsburg
    Jan 16, 2014 at 20:17
  • Just the ssh connection. and the checks that need that connection to work - like check_remote_disk (since we are using connect_by_ssh instead or NRPE) Jan 16, 2014 at 20:23
  • So it sounds like you have multiple services failing, and they're all using check_by_ssh, correct? I didn't think there was a connect_by_ssh Also, with check_by_ssh, the below has worked for me with intermittent failures in the past. Adding the -E to the check_by_ssh ignores the stderr output. It may be hanging on something. -S ignore all stdout and just returns data as well. Also, add the -v option to the check_by_ssh command.
    – rfelsburg
    Jan 16, 2014 at 20:27
  • you're right, its check_by_ssh, my mistake, a bit under pressure here, lol. The -E is already there, i'll add -v and -S. Thanks :) Jan 16, 2014 at 20:35
0

I've seen this before as a DNS issue.

Perhaps the rDNS lookup times out (as noted in comments above) or perhaps the server is actually several servers using round-robin DNS (multiple A records for one domain name) and only a subset of the servers is offline, not running SSH, or otherwise fails the test.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .