SSH Agent forwarding not working on Ubuntu 22.04

Question

I am trying to connect to github over ssh on my remote server (Running Ubuntu 22.04).

On my local computer (Running Win 10), I have ~/.ssh/config file with the following:

Host remote
    HostName SERVER_IP
    port 22
    User ubuntu
    ForwardAgent yes

After connecting to the remote server, I can confirm that the ssh agent is working by typing:

echo "$SSH_AUTH_SOCK"

result: /tmp/ssh-XXXXPWEKZo/agent.1073

Also with ssh-add -l I see that the key is added

4096 SHA256:hvGuLtIuwYi2LAnQ0KdC/9IgdBUmlHZer0NyXUXd5aY C:\Users\user/.ssh/id_rsa (RSA)

In /etc/ssh/sshd_config I have allowed Agent forwarding with AllowAgentForwarding yes

But when I try to connect to github ssh -T [email protected] -vvv (the key is added to github settings) I get the following:

OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/ubuntu/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/ubuntu/.ssh/known_hosts2'
debug2: resolving "github.com" port 22
debug3: resolve_host: lookup github.com:22
debug3: ssh_connect_direct: entering
debug1: Connecting to github.com [140.82.121.3] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/ubuntu/.ssh/id_rsa type -1
debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/ubuntu/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519_sk type -1
debug1: identity file /home/ubuntu/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_xmss type -1
debug1: identity file /home/ubuntu/.ssh/id_xmss-cert type -1
debug1: identity file /home/ubuntu/.ssh/id_dsa type -1
debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
debug1: Remote protocol version 2.0, remote software version babeld-2f5f2727
debug1: compat_banner: no match: babeld-2f5f2727
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to github.com:22 as 'git'
debug3: record_hostkey: found key type ED25519 in file /home/ubuntu/.ssh/known_hosts:1
debug3: load_hostkeys_file: loaded 1 keys from github.com
debug1: load_hostkeys: fopen /home/ubuntu/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type [email protected], using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,[email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-ed25519,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
debug2: MACs stoc: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly[email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
debug3: record_hostkey: found key type ED25519 in file /home/ubuntu/.ssh/known_hosts:1
debug3: load_hostkeys_file: loaded 1 keys from github.com
debug1: load_hostkeys: fopen /home/ubuntu/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'github.com' is known and matches the ED25519 host key.
debug1: Found key in /home/ubuntu/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
channel 1: chan_shutdown_read: shutdown() failed for fd 7 [i0 o0]: Not a socket
debug2: get_agent_identities: ssh_agent_bind_hostkey: communication with agent failed
debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed
debug1: Will attempt key: /home/ubuntu/.ssh/id_rsa
debug1: Will attempt key: /home/ubuntu/.ssh/id_ecdsa
debug1: Will attempt key: /home/ubuntu/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/ubuntu/.ssh/id_ed25519
debug1: Will attempt key: /home/ubuntu/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/ubuntu/.ssh/id_xmss
debug1: Will attempt key: /home/ubuntu/.ssh/id_dsa
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ubuntu/.ssh/id_rsa
debug3: no such identity: /home/ubuntu/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa
debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa_sk
debug3: no such identity: /home/ubuntu/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519
debug3: no such identity: /home/ubuntu/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519_sk
debug3: no such identity: /home/ubuntu/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_xmss
debug3: no such identity: /home/ubuntu/.ssh/id_xmss: No such file or directory
debug1: Trying private key: /home/ubuntu/.ssh/id_dsa
debug3: no such identity: /home/ubuntu/.ssh/id_dsa: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).

I don't have such problems on other of my servers that are running ubuntu 20.04 and 18.04.

Answer 1

This answer helped me solve the same problem: there's some incompatibility between the ssh client shipped with Windows and the server on Ubuntu 22.04.

tldr: Installing the most recent release of the windows ssh client fixed the issue for me.

Answer 2

The answer from @kaytwo is right; updating the Windows ssh client is the cure.

Here's what strace -tt -f -v -s1024 ssh $hostname revealed:

2636504 12:05:25.582068 write(3, "\0\0\0\320", 4) = 4
2636504 12:05:25.582159 write(3, "\33\0\0\0\[email protected]\0\0\0003\0\0\0\vssh-ed25519\0\0\0.....binary-data-omitted.....", 208) = 208
2636504 12:05:25.582210 read(3, "", 4)  = 0
2636504 12:05:25.875257 getpid()        = 2636504
2636504 12:05:25.875343 write(3, "\0\0\0\1", 4) = 4
2636504 12:05:25.875532 write(3, "\v", 1) = 1
2636504 12:05:25.875691 read(3, "", 4)  = 0
2636504 12:05:25.875745 getpid()        = 2636504
2636504 12:05:25.875799 write(2, "debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed\r\n", 87) = 87

Compare with strace -tt -f -v -s1024 ssh-add -l:

2648504 13:00:38.631807 connect(3, {sa_family=AF_UNIX, sun_path="/tmp/ssh-XXXXuWGsZN/agent.2628678"}, 110) = 0
2648504 13:00:38.632285 write(3, "\0\0\0\1", 4) = 4
2648504 13:00:38.632386 write(3, "\v", 1) = 1
2648504 13:00:38.632464 read(3, "\0\0\1\255", 4) = 4
2648504 13:00:38.920854 read(3, "\f\0\0\0\1\0\0\1\227\0\0\0\7ssh-rsa\0\0\0\3\1\0\1\0\0\1...binary-data-omitted.....", 429) = 429

Basically, sending a "[email protected]" request causes the Windows ssh-agent to stop talking; the normal "list all keys" command doesn't return anything when it's in that state. Fortunately, the state is cleared when the connection is closed, so you can try again without having to restart anything.

It looks like this configuration option (in man ssh_config) is related:

 PubkeyAuthentication
        Specifies whether to try public key authentication.  The argument to this keyword must
        be yes (the default), no, unbound or host-bound.  The final two options enable public
        key authentication while respectively disabling or enabling the OpenSSH host-bound au‐
        thentication protocol extension required for restricted ssh-agent(1) forwarding.

Note: While that paragraph says ssh -o PubkeyAuthentication=unbound ..... should disable this protocol extension, strace showed that it was still being sent, and the Windows ssh agent won't work.

This page on SSH Agent Restriction goes into total detail about the subject, from the reasons that inspired the feature to command-line options needed to use it and the byte sequences sent in the SSH Agent protocol extension.

Good luck!

Answer 3

This is already answered but If you are here to look for a solution for an error like chan_shutdown_read: channel 1: shutdown() failed for fd 7 [i0 o0]: Not a socket

This only applies if you have installed latest version of OpenSSH from GitHub and your forwarding agent suddenly stopped.

In my case after Windows update Forwarding Agent suddenly stopped working. I could connect the server but since my SSH key is not forwarded I could not able to push or pull from GitHub Private Repo using SSH and It was saying access denied for public key.

To confirm if the solution is same: Open services.msc and find OpenSSH Authentication Agent. Double click to open it's properties and check Path to executable section. If it is not "C:\Program Files\OpenSSH\ssh-agent.exe" and pointing to ssh agent in system32 than this solution will work.

open regedit and head to HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/ssh-agent Change ImagePath value to "C:\Program Files\OpenSSH\ssh-agent.exe" and restart the computer.

After restart make sure that OpenSSH Authentication Agent service is running then add your SSH key by typing ssh-add in PowerShell or CMD. Then your Forwarding should work.