Connection closed by UNKNOWN port 65535 when SSH using AD Creds on RHEL Machine

Question

Recently I installed PAM & all the necessary packages needed to enable ssh authentication via AD on my RHEL 7.5 machines.

When I try to SSH using "ssh user@domainname@hostname" it asks for my password and as soon as I type my password, I get the error: Connection closed by UNKNOWN port 65535.

This happens on several machines running RHEL 7.5. sssd is used to bind to AD, there are no firewalls, system time is in sync with AD server, appropriate groups are added to permitted_groups, and the account is not locked out. Tailing the /var/log/secure shows the errors below:

sshd[12752]: reprocess config line 145: Deprecated option RhostsRSAAuthentication
sshd[12752]: pam_krb5[12752]: account checks fail for 'user@domainname': user disallowed by .k5login file for 'user@domainname'
sshd[12752]: Failed password for user@domainname from ip port 53166 ssh2
sshd[12752]: fatal: Access denied for user user@domainname by PAM account configuration [preauth]

I have restarted sssd and sshd without any luck.

Any help will be appreciated.

Answer 1

The super-unhelpful ssh error Connection closed by UNKNOWN port 65535 can be reported when your ssh client in a couple of different situations when the remote sshd cannot be reached at all because of something happening "in the middle".

This can be extra-tricky to debug because in some cases the remote sshd has no idea that anyone is ever tried to connect to it.

(Aside 65535 is "special" number to computer folks as it is 2^16 - 1, aka 0xFFFF -- the maximum unsigned 16 bit integer (also the max port number))

Case A -- (From @doug 's original question) - In this case the remote sshd got the incoming connection and delegated auth down to Linux libraries for PAM (Pluggable Authentication Modules). PAM hands off to KRB5 or SSS and that fails. So all the poor remote sshd gets is a big NOPE from PAM. ...it never got into it's "normal" protocol parsing and error checking that would let it return a more helpful error message.

(It's possible that old Kerberos config options like gssapiauthentication might behave similarly)

Case B -- In our case we saw this when network firewalls prevented connections from the dev/test machines to staging/production machines. Depending on your network, you might be able get more diagnostics info with tcping 22 $remote_hostname, or (less helpful): UDP network tests like ping $remote_hostname, traceroute $remote_hostname, or the IPv6 versions of those commands. Your local network engineers can help confirm & fix.

The giveaway in this case is that ssh -vvv $remote_hostname gets to this point:

debug1: identity file /home/ddickinson/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6

...pauses for 60s (or whatever timeout), then:

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Case C -- Some kinds of failures of the ProxyCommand that your local ssh delegates to can also fail in unhelpful ways. Check for any "proxy*" or "tunnel*" related options in the output of:

ssh -G $remote_hostname