SSH through intermediate host fails only on myuser@mymac but works elsewhere

Question

I am not able to connect via ssh to one of my datacenter nodes using my user on my macbook. This is a recent problem, and it was perfectly funtional since ~ a couple of weeks ago.

Strangely, this only affects my user on my computer, but I am able to establish the connection from:

• A different user in the same machine, using the same ssh keys and without any .ssh/config rule.
• A different server, running either macos or ubuntu, with the same or different ssh keys.

Using my username in my computer, and same keys, I can:

• Connect to the gateway host
• Connect directly to the node using a VPN (unfortunately this is not a long-time solution)

I’m quite puzzled by this error. Can you help me locate the problem?

Looking at the logs, the connection with the gateway is established, but somehow fails at connecting to the node. On the client side:

⌘ ~ ❯ ssh -v -J gatekeeper@gateway ubuntu@node -i ~/.ssh/id_rsa 
OpenSSH_7.3p1, LibreSSL 2.4.1
[...]
debug1: Authentication succeeded (publickey).
Authenticated to gateway ([35.156.248.245]:22).
debug1: channel_connect_stdio_fwd node:22
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 1
debug1: client_input_global_request: rtype [email protected] want_reply 1
debug1: client_input_global_request: rtype [email protected] want_reply 1
debug1: client_input_global_request: rtype [email protected] want_reply 1
debug1: client_input_global_request: rtype [email protected] want_reply 1
debug1: client_input_global_request: rtype [email protected] want_reply 1
debug1: client_input_global_request: rtype [email protected] want_reply 1
debug1: client_input_global_request: rtype [email protected] want_reply 1
channel 0: open failed: connect failed: Connection timed out
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

On the gateway side:

admin@gateway:~$ grep -e "\[7669\]" -e "\[7739\]" /var/log/auth.log
Mar 13 11:01:20 gateway sshd[7669]: Set /proc/self/oom_score_adj to 0
Mar 13 11:01:20 gateway sshd[7669]: rexec line 32: Deprecated option PermitBlacklistedKeys
Mar 13 11:01:20 gateway sshd[7669]: Connection from <laptop-out-ip> port 62113 on <gateway-ip> port 22
Mar 13 11:01:20 gateway sshd[7669]: Postponed publickey for gatekeeper from <laptop-out-ip> port 62113 ssh2 [preauth]
Mar 13 11:01:20 gateway sshd[7669]: Accepted publickey for gatekeeper from <laptop-out-ip> port 62113 ssh2: RSA 8d:7e:9c:53:11:c9:4d:b3:67:7b:ae:04:03:8f:e2:71
Mar 13 11:01:20 gateway sshd[7669]: pam_unix(sshd:session): session opened for user gatekeeper by (uid=0)
Mar 13 11:01:20 gateway sshd[7669]: User child is on pid 7739
Mar 13 11:03:27 gateway sshd[7739]: error: connect_to <node-ip> port 22: failed.
Mar 13 11:03:28 gateway sshd[7739]: Connection closed by <laptop-out-ip>
Mar 13 11:03:28 gateway sshd[7739]: Transferred: sent 2252, received 2864 bytes
Mar 13 11:03:28 gateway sshd[7739]: Closing connection to <laptop-out-ip> port 62113
Mar 13 11:03:28 gateway sshd[7669]: pam_unix(sshd:session): session closed for user gatekeeper

On the node side there is no entry in the logs.

The ssd_config at the gateway:

# ssh service configuration

AcceptEnv
AddressFamily inet
AllowAgentForwarding yes
AllowGroups
AllowTcpForwarding no
AllowUsers gatekeeper
AuthorizedKeysFile %h/.ssh/authorized_keys
ChallengeResponseAuthentication no
Ciphers aes128-ctr,aes192-ctr,aes256-ctr
ClientAliveCountMax 3
ClientAliveInterval 15
Compression delayed
DenyGroups
DenyUsers
GSSAPIAuthentication no
GatewayPorts no
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostbasedAuthentication no
KerberosAuthentication no
ListenAddress 0.0.0.0:22
LogLevel VERBOSE
LoginGraceTime 60
MaxAuthTries 6
MaxSessions 10
MaxStartups 30
PasswordAuthentication no
PermitBlacklistedKeys no
PermitRootLogin no
PermitTunnel no
PermitUserEnvironment no
PidFile /var/run/sshd.pid
PrintLastLog yes
PrintMotd no
Protocol 2
PubkeyAuthentication yes
RSAAuthentication no
RhostsRSAAuthentication no
StrictModes yes
SyslogFacility AUTH
TCPKeepAlive yes
UseDNS no
UseLogin no
UsePAM yes
UsePrivilegeSeparation yes
X11Forwarding no

Match User gatekeeper
AllowTcpForwarding yes
AllowAgentForwarding no
X11Forwarding no

Answer 1

Finally I have been able to corner and identify the source of the problem. I could make the problem disappear by not sourcing the iterm2 shell integration, or just updating it to the latest version. This may be somehow related to using the fish shell.

I did not dig further into the problem, if anyone's interested please let me know.

Answer 2

error: connect_to <node-ip> port 22: failed.

This looks like the port forwarding is disabled or limited by PermitOpen option in sshd config. Make sure it is allowed, if not, please post the sshd_config from gateway.

Answer 3

that I am able to connect with the same command, keys, etc, from the same macbook and a different user

Maybe you need to RUN this with the new user (in your home directory):

ssh-keygen -t rsa

ssh-copy-id -i .ssh/id_rsa.pub gatekeeper@gateway 

ssh-copy-id -i .ssh/id_rsa.pub ubuntu@node

Answer 4

I had something similar to this. I wasn't able to ssh through the intermediate host, but I was able to ssh into the intermediate host.

Turns out I just had an outdated ssh/config. Overwrote my ssh/config with the ssh_proxy_config that was on the intermediate host, and I was good to go

Answer 5

I hit almost same problem and in my case ssh -v target emitted

(... while connecting to jumphost.example.com:)
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:REDACTED /home/USER/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-512 blen 277
debug1: Authentication succeeded (publickey).
Authenticated to jumphost.example.com ([1.2.3.4]:22).
debug1: channel_connect_stdio_fwd target:1234
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 0
channel 0: open failed: connect failed: Connection refused
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

So similar up to debug1: pledge: network. After that I see hostkey-00 instead of keepalive and channel 0: open failed: connect failed: Connection refused.

In my case it was a typo in my .ssh/config where Port of Host target was accidentally set to 1234 instead of the correct port.

Unfortunately, the diagnostics of ProxyJump or ssh -J are seriously lacking. The feature itself is superb, though!

Answer 6

i had the problem

ssh_exchange_identification: Connection closed by remote host

in my case the problem was that i tried to ssh "B" via ssh "A" and used incorrect "B" ip. "B" ip didnt exist.

for debug i suggest to use client-side debugging as "ssh proxy" side debugging just useless. sshd doesnt log anything on ssh proxy host even if we successfully logon to "B".

client side debugging looks like this:

$ ssh -vvv "B"

when using ssh tunnel i suggest to use "ProxyJump" it gives lots of usefull info on client side.

i do not recommend using "ProxyCommand" . it gives nothing when debugging trouble connection.

typical ssh proxy config

Host nl-test-05  172.16.10.21
HostName 172.16.10.21
IdentityFile ~/.ssh/vasya
ProxyJump nl-proxy-01-test
Port 22
User vasya
ForwardAgent yes 

Answer 7

I had this same problem (able to SSH to a bastion, but not able to SSH through it).

In my case it was because I'd used a ControlPath statement in my .ssh/config for the bastion host when trying other things out. I removed that and it all worked again.

Possibly because I'm using Windows and a few years ago at least, it wasn't supported.