layerd ssh connections - second ssh command looks for keys in wrong directory

Question

I'm logged into a server called Walnut, from which I try to log into another server called Hazelnut.

local machine (mac) ---ssh---> Walnut ---ssh---> Hazelnut

The first ssh (from my local machine to Walnut) goes fine.

The second ssh command, however, gives me permission denied.

Here's what the logs say when I do ssh -v -A Haezlnut.

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to Hazelnut [address] port [port].
debug1: Connection established.
debug1: identity file /home/username/.ssh/id_rsa type -1
debug1: identity file /home/username/.ssh/id_rsa-cert type -1
debug1: identity file /home/username/.ssh/id_dsa type -1
debug1: identity file /home/username/.ssh/id_dsa-cert type -1
debug1: identity file /home/username/.ssh/id_ecdsa type -1
debug1: identity file /home/username/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u2
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u2
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: [Host key]
debug1: Host 'Hazelnut' is known and matches the ECDSA host key.
debug1: Found key in /home/username/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/username/.ssh/id_rsa
debug1: Trying private key: /home/username/.ssh/id_dsa
debug1: Trying private key: /home/username/.ssh/id_ecdsa
debug1: No more authentication methods to try.
Permission denied (publickey).

So I think the logs clearly indicate that it's failing to find the key file at the end. What's confusing is that since the private key resides in the local mac machine, it should look for the file in /Users/username/.ssh/, not /home/username/.ssh/. The first ssh command (from local to Walnut) does this fine, but the second command (from Walnut to Hazelnut) somehow messes it up.

What's more confusing is that the exact same process works perfectly fine on many other seemingly identical mac machines. If you try to ssh into Hazelnut from Walnut using any other mac machine, it attempts to look for the key file in the correct (/Users/username/.ssh) directory.

Has anyone had this issue before?

Answer 1

You'd need to either make your ssh agent is available on walnut using this command from your mac: ssh -A walnut then ssh on to hazelnut.

Doing this kind of thing though is thought of as bad practice as it exposes your agent to the remote machine. Should an attacker be on walnut its possible they could steal your private keys.

Or tunnel to hazelnut. There are many ways to do this though...

Use the -J option - a shortcut for the ProxyJump option ( openssh ver. 7.3+ Required ):

ssh -J <jumphost> <target>

The ProxyCommand option using netcat on the jump host:

ssh -o ProxyCommand="ssh %h nc <target> 22" <jumphost>

or the ProxyCommand with the -W option:

ssh -o ProxyCommand="ssh -W %h:%p <jumphost>" <target>

The proxycommand or ProxyJump configuration directives could also be placed in the ssh configuration file for the same effect