SSH config ProxyCommand asks for public key

Question

I'm trying to connect via a gateway to an EC2 instance.

If I connect to gateway

local> ssh gateway

Then I can connect to EC2 without a password

gateway> ssh ec2  # works

However, trying to connect through the proxy seems to require the indentity file.

Host gateway
    HostName <gateway>

Host ec2
    HostName ec2-<ec2>.compute.amazonaws.com
    ProxyCommand ssh gateway -W %h:%p


local> ssh ec2
Permission denied (publickey).

I thought the ProxyCommand is basically logging me into the gateway and then logging into the final destination. If so, why would it ask me for the public key when the gateway is setup to not require it? How can I connect to the ec2 instance in the same manner as if I ssh'd into the gateway and then ssh'd into ec2?

Edit: rsync works normally (doesn't ask for the key file)

rsync -pthrvz  --rsync-path=/usr/bin/rsync --rsh='ssh gateway ssh' . ec2:/path

ssh -v output from gateway> ssh ec2

gateway> ssh -v ec2
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to ec2 port 22.
debug1: Connection established.
debug1: identity file <snip> type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'ec2' is known and matches the RSA host key.
debug1: Found key in ~/.ssh/known_hosts:63
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: ~/.ssh/identity
debug1: Offering public key: ~/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.

ssh -v output from trying to connect via proxy

local$ ssh -v ec2 
OpenSSH_6.7p1 Ubuntu-5ubuntu1.3, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /home/matt/.ssh/config
debug1: /home/matt/.ssh/config line 9: Applying options for ec2
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Hostname has changed; re-reading configuration
debug1: Reading configuration data /home/matt/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Executing proxy command: exec ssh gateway -W ec2:22
debug1: permanently_drop_suid: 1000
debug1: identity file /home/matt/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/matt/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/matt/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/matt/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/matt/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/matt/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/matt/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/matt/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Ubuntu-5ubuntu1.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 73:<snip>:0c
debug1: Host 'ec2.compute.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /home/matt/.ssh/known_hosts:11
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/matt/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/matt/.ssh/id_dsa
debug1: Trying private key: /home/matt/.ssh/id_ecdsa
debug1: Trying private key: /home/matt/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

The problem seems to be that it's trying to find the key on my machine rather than taking it from the gateway machine.

Answer 1

ProxyCommand doesn't work the way you think it does. The command specified is not run on the gateway machine. Rather, it is run on the connecting machine.

The flow of execution is therefore:

1. The ProxyCommand "ssh gateway -W %h:%p" is executed on the "local" machine. This establishes an SSH session to the gateway, using the RSA identity on your local box. The -W flag specifies that stdin and stdout are to be hooked up to a TCP session origination on the gateway to your final destination.

2. With the proxy session established, ssh on your local box again uses that session to authenticate to your remote SSH server, again, using local credentials.

It's a bit confusing, but think of the ProxyCommand as simply setting up a "pipe" between your local SSH client and the SSH server that is your final destination. That dumb pipe is then used by your local SSH client to talk to the final destination's SSH service.

The key is that there are therefore two instances of SSH running on your local box, one of them being the ProxyCommand, the other the actual SSH connection you want to establish! You should be able to verify that by looking at the output of "ps aux" on your local box.

That would then explain why it's trying to use key material on your local box, rather than on your gateway to authenticate. :-)

The reason rsync works, is that you're actually doing "ssh gateway ssh" as your --rsh command, which actually runs ssh once on the local box to connect to the gateway, and then once again on the remote box, which will then use the remote key material.

Hope this helps.