5

(previously posted at stackoverflow by error)

I'm running a bunch of servers with Ubuntu 14.04.1 (sun,hyperion,...) all of which use public keys (OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 on all machines) for rsync without problems. Almost all...

One connection fails without any changes in the configs or keys. Then I'll try to re-add the keys, check for ECDSA, reboot/restart ssh and it works again. Or it doesn't. In this case I just wait a random amount of time (1h up to 3 months) an do the same. This time it fixes the problem - for a while.

The relevant parts of a ssh -vvv diff:

Successful connection

debug1: Host 'hyperion.internal' is known and matches the ECDSA host key.
debug1: Found key in /home/bar/.ssh/known_hosts:20
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/bar/.ssh/id_rsa (0x7f..),
debug2: key: /home/bar/.ssh/id_dsa ((nil)),
debug2: key: /home/bar/.ssh/id_ecdsa ((nil)),
debug2: key: /home/bar/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 95:...
debug3: sign_and_send_pubkey: RSA 95:...
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to hyperion.internal ([172.16.0.10]:22).

Failed connection

debug1: Host 'hyperion.internal' is known and matches the ECDSA host key.
debug1: Found key in /home/bar/.ssh/known_hosts:20
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/bar/.ssh/id_rsa (0x7f..),
debug2: key: /home/bar/.ssh/id_dsa ((nil)),
debug2: key: /home/bar/.ssh/id_ecdsa ((nil)),
debug2: key: /home/bar/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/bar/.ssh/id_dsa
debug3: no such identity: /home/bar/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/bar/.ssh/id_ecdsa
debug3: no such identity: /home/bar/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/bar/.ssh/id_ed25519
debug3: no such identity: /home/bar/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

Things that I've checked several times:

  • permissions for .ssh/ and id_rsa on all machines
  • that I'm using the right keys
  • that ssh-copy-id -i /home/bar/.ssh/id_rsa europa@hyperion.internal copies the right keys to the right authorized_hosts file

What didn't really help but added to the vodoo/heisenbug effect:

  • rebooting the machines
  • restarting the ssh service
  • fiddling with global ssh options

I've pasted the full logs with some redacted info at pastebin: wall of log

1
  • Anything in /var/log/secure (or whatever Ubuntu's equivalent is?) – Aaron Copley Oct 31 '14 at 15:39
3

The problem has been resolved, it wasn't ssh-related at all:

hyperion.internal had an encrypted home, so the key-lookup failed when it was not mounted to /home/europe.

In hindsight quite obvious, but it accounts for the heisenbug effect of not failing when observing the logs on the machine (while being logged in, of course...)

Hope this helps at least some else.

2
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password

This indicates the server didn't accept your private key. Unfortunately, the server doesn't provide the client with any more detail about why it didn't accept the key, so you really need to troubleshoot this on the server.

I would start by checking the syslogs in /var/log on the server for any messages from sshd indicating why it rejected the authentication attempt.

If you have root access on the remote server, you can run a debugging instance of sshd and then connect to it with a client. On the remote server, become root and run /path/to/sshd -d -p 2222. This will launch an instance of sshd which listens on port 2222. It will accept one connection, and it will print debugging information to your terminal.

Then, on the client, run ssh as normally but include -p 2222 to connect to the correct port. If the login fails, check the debugging output printed by the server.

2
  • Here's the funny part. I did this and I can log in without password! Switch back to default port, and I can't any more. So it seems the server-side sshd (on default port) is not looking at the authorized_keys file, but the test sshd on 2222 is doing so. I already tried service sshd restart but it doesn't help. How do I force sshd to reload or look at authorized_keys file? – ADTC Dec 11 '15 at 8:25
  • 1
    HA! Found the solution (for CentOS at least). I had to run restorecon -R -v /root/.ssh on the server side, and it spewed two lines as though it's fixing something. (I ran it again, and it didn't spew anything.) (Note: Previously I had tried applying correct permissions on .ssh, home folder, authorized_keys etc as per various tips online, but it wasn't enough.). The output was something like: restorecon reset /root/.ssh/authorized_keys context unconfined_u:object_r:admin_home_t:s0->unconfined_u:object_r:ssh_home_t:s0 – ADTC Dec 11 '15 at 8:36
2

For me, this was a permissions issue involving the Home directory as well. The permissions for the Home directory on the destination server was set to 775. From what I discovered, the Home directory permissions must be set to 755 or less. This sets it to where no user other than the owner of the home directory is allowed to have write permissions.

1

It looks like a server issue:

debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password

The server does not seem to be sending a reply here. I would try cranking the debug up to 11 on the server side and see what it is whining about.

How much time elapses after the publickey packet is sent waiting for the reply when it fails? If it i

1
  • Actually, the 'Authentications that can continue' message is logged when the client receives an auth fail message from the server. – Kenster Nov 3 '14 at 20:25
0

Chown the root-created files (not just chmod) back to that original user account. You could also try testing with Userify, see if it works and that you have public key without errors.

0

I've often had ssh problems due to the same MAC address being associated with multiple computer names and visa-versa. There are command-line options for ssh to handle this. You can also blow away or edit your known-hosts file to eliminate the problem. Not sure if this helps but this covers 90% of my ssh problems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.