Why do I get a ssh Permission denied (publickey)?

Question

I have two machines on Google compute engine and I want to ssh the other server. Therefore, on the first machine (namenode), I created a passwordless ssh key for user hadoop and did a cat id_rsa.pub >> authorized_keys. On the second machine (datanode1) there is also a user called hadoop. The second machine's hadoop user has an empty ~/.ssh directory.

When I now try to ssh from the first to the second machine I get:

hadoop@namenode:~/.ssh$ ssh -v hadoop@datanode1
OpenSSH_6.0p1 Debian-4, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to datanode1 [10.240.223.55] port 22.
debug1: Connection established.
debug1: identity file /home/hadoop/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/hadoop/.ssh/id_rsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_dsa type -1
debug1: identity file /home/hadoop/.ssh/id_dsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_ecdsa type -1
debug1: identity file /home/hadoop/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-4
debug1: match: OpenSSH_6.0p1 Debian-4 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 10:54:3e:ec:07:58:48:85:28:40:90:43:e1:8f:3d:f0
debug1: Host 'datanode1' is known and matches the ECDSA host key.
debug1: Found key in /home/hadoop/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/hadoop/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/hadoop/.ssh/id_dsa
debug1: Trying private key: /home/hadoop/.ssh/id_ecdsa
debug1: No more authentication methods to try.
Permission denied (publickey).

I do not understand what is happening here.

UPDATE: I did copy the id_rsa.pub to the authorized_keys file on the destination server. However, now I get the following error:

hadoop@namenode:~/.ssh$ ssh datanode1
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for datanode1 has changed,
and the key for the corresponding IP address 10.240.226.88
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
9f:8d:07:09:a9:67:63:b4:b9:2b:f5:39:ed:ef:55:d6.
Please contact your system administrator.
Add correct host key in /home/hadoop/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/hadoop/.ssh/known_hosts:1
ECDSA host key for datanode1 has changed and you have requested strict checking.
Host key verification failed.

Adding "StrictHostKeyChecking=no" to the /etc/ssh/ssh_config repaired this issue.

Answer 1

Your SSH public key needs to be copied to the destination server, not the source.

• Create your SSH key on the server you will be connecting from
• Cat the ~/.ssh/id_rsa.pub file you just created
• Copy this key to the ~/.ssh/authorized_keys file on the sever you will be connecting to
• Check permissions on ~/.ssh/authorized_keys are set to 0600
• You can also use the ssh-copy-id helper application depending on your OS (I don't have direct experience with Google Compute, so this may not work for you).

A good way (for me at least) of remembering how SSH key authentication works is this:

• The computer you're connecting from reads the private key (e.g. ~/.ssh/id_rsa), and generates the public key to forward to the receiving machine.
• The key file (e.g. ~/.ssh/id_rsa.pub) that is generated when creating the key is for information only, and can be re-created/calculated if you have the private key.
• Because of this, your private key should never be shared.
• The ~/.ssh/authorized_keys file is a "white-list" of sorts. It lists all the public key signatures that are allowed to connect to that user's account.

Answer 2

i'd suggest you to update the host fingerprint in /home/hadoop/.ssh/known_hosts (or delete the line and re-confirm the host when logging in again) and not disable the StrictHostKeyChecking.

(this assuming that your server doesn't change IP maintaining the DNS)