1

We have a dev server running jenkins with various jobs that run shell scripts that connect to our live server via SSH (using key-based authentication).

This doesn't seem to work reliably.

If I test the SSH connection on its own by logging in to the dev server as the "jenkins" user and then SSH'ing to the live server, the authentication works and logs me in without the password prompt.

If I run one of the jenkins jobs manually it works most of the time (although one failed this morning prompting me to write this ticket) but if I automate the job to run nightly it fails most of the time (I think it has worked once).

The point of failure each time is at the SSH connection:

> ssh -T nepton@xxx
> Permission denied, please try again.
> Permission denied, please try again.
> Permission denied (publickey,password).

Any help would be greatly appreciated as I'm completely stuck and Linux is definitely not my area of expertise.

Additional Information:

  • I generated the public key on Dev (whilst logged in as "jenkins" user) using: ssh-keygen -t rsa
  • I copied the key to the live server using: ssh-copy-id -i ~/.ssh/id_rsa.pub nepton@xxx

Permissions of /var/lib/jenkins/.ssh/ on Dev are:

drwx------  2 jenkins jenkins 4096 Apr  4 13:31 .
drwxr-xr-x 13 jenkins jenkins 4096 Apr  4 13:38 ..
-rw-------  1 jenkins jenkins 1679 Apr  4 13:31 id_rsa
-rw-r--r--  1 jenkins jenkins  400 Apr  4 13:31 id_rsa.pub
-rw-r--r--  1 jenkins jenkins 1338 Apr  4 13:33 known_hosts

Permissions of /home/nepton/.ssh/ on the Live server are:

drwx------ 2 nepton nepton 4096 Mar 20 17:37 .
drwx------ 4 nepton nepton 4096 Apr  2 10:52 ..
-rw------- 1 nepton nepton 2005 Apr  8 10:23 authorized_keys
-rw-r--r-- 1 nepton nepton 1106 Apr  1 10:39 known_hosts
  • I'm running the latest LTS version of Ubuntu and the standard SSH package on both servers.

Debug information

ssh -v -T -i /var/lib/jenkins/.ssh/id_rsa nepton@xxx
OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to xxx [xx.xxx.xx.xxx] port 22.
debug1: Connection established.
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA xxx
debug1: Host 'xxx' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:7
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/jenkins/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
debug1: No more authentication methods to try.
Permission denied (publickey,password).
Build step 'Execute shell' marked build as failure
9
  • Could you check your server logs if you find something. DNS issue ?
    – Dom
    Apr 8, 2013 at 9:13
  • Edit the scripts to use ssh -v -T nepton@xxx. That will give you a lot more information; add the output to your question.
    – Jenny D
    Apr 8, 2013 at 9:16
  • Make sure you make the first connection logged in as jenkins, that way you can confirm the acceptance of the key.
    – NickW
    Apr 8, 2013 at 9:18
  • @Dom I had a look in /var/log/syslog and /var/log/jenkins and found nothing useful Apr 8, 2013 at 9:58
  • 1
    Yeah, I've tripped over the accept server key (known hosts file) a few times in the past.
    – NickW
    Apr 8, 2013 at 10:05

2 Answers 2

1

As you say in comments, it's not using the key you want it to. You can point out the correct key like this:

ssh -T -i /path/to/real/id_rsa nepton@xxx
1
  • This worked the day I tried it but no longer works today. I have updated the debug information in my original post. Thanks. Apr 11, 2013 at 6:53
1

I had this same issue and found a solution that may be related here. If the system you are attempting to SSH to is behind a load balancer (or something similar) it's possible that sometimes your SSH can succeed, and sometimes it will fail if the machines that are being load balanced do not have a shared home directory (or at least your .ssh information such as the public key).

From your side, it will look like you are logging into a single host, but actually it is intermittently a different actual host, which could explain this odd behavior.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .