Connection to one of my servers using ssh takes more than 20 seconds to initiate.

This is not related to LAN or WAN conditions, since connection to itself takes the same (ssh localhost). After connection is finally establised, it is super fast to interract with the server.

Using -vvv shows that the connection is stuck after saying "pledge: network". At this point, authentication (here using key) is already done, as visible here :

...
debug1: Authentication succeeded (publickey).
Authenticated to myserver.mydomain.com ([xx.xx.xx.xx]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network

(...stuck here for 15 to 25 seconds...)

debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
...

Server is Ubuntu 16.04. It already happened to me in the past with another server (was Ubuntu 12.04) , nerver found the solution and the problem disapeared after a while...

sshd_config is the default one provided by Ubuntu.

So far I have tried :

  • using -o GSSAPIAuthentication=no in the ssh command
  • using password instead of a key
  • using UsePrivilegeSeparation no instead of yes, in sshd_config
  • 1
    Usually for me slow SSH connections are DNS problems, might that be the case here? For example, the server may be stuck trying to do a reverse DNS for the client's IP and waiting for that to time out – Eric Renouf Jul 28 '16 at 14:09
  • Actually no : by default UseDNS is not defined in sshd_config and man page says that this option is "no" by default. – M-Jack Jul 28 '16 at 15:17
  • 2
    Some Googling suggests this can be caused by updating systemd without rebooting. And there was a systemd update for xenial on July 12. systemctl restart systemd-logind fixes the problem only for a short period of time for me. – Ivan Kozik Aug 15 '16 at 18:19
  • Or if you're seeing pam_systemd(sshd:session): Failed to create session: Connection timed out as mentioned in an answer, this might be github.com/systemd/systemd/issues/2925 – Ivan Kozik Aug 15 '16 at 19:17
  • I came here having had this problem after an update, and @IvanKozik's suggestion fixed the problem - i.e systemctl restart systemd-logind - so thanks for that. – Paul M Nov 23 '16 at 23:45

This is probably an issue with D-Bus and systemd. If the dbus service is restarted for some reason, you will also need to restart systemd-logind.

You can check if this is the issue by opening the ssh daemon log (on Ubuntu it should be /var/log/auth.log) and check if it has these lines:

sshd[2721]: pam_systemd(sshd:session): Failed to create session: Connection timed out

If yes, just restart systemd-logind service:

systemctl restart systemd-logind

I had this same issue on CentOS 7, because the messagebus was restarted (which is how the D-Bus service is called on CentOS).

  • I tried to restart systemd-logind but after a while it says PolicyKit daemon disconnected from the bus. We are no longer a registered authentication agent. Job for systemd-logind.service failed because a timeout was exceeded. See "systemctl status systemd-logind.service" and "journalctl -xe" for details. – Kun Ren May 12 '17 at 3:49
  • @KunRen you probably need to restart the polkit service using systemctl restart polkit. – Strahinja Kustudic Jun 27 '17 at 18:57
  • This worked for me, thank you :) – Wolfe Sep 5 at 19:26

found the answer :

changed UsePAM from yes to no in sshd_config file

After restarting the ssh service, the connection is now immediate to the server. On this server, PAM is linked to ldap, so that is probably the reason, even if here I am connecting with a user declared on the server itself, not an LDAP one.

Well, this is more a way to bypass the issue, not really a solution... I have other servers set up the same way that are not having this issue.

Hope this may help someone...

  • 1
    changing UsePAM to no has other effects. See this discussion So I had to set a password to the user, because I got errors like User nagios not allowed because account is locked – M-Jack Jul 28 '16 at 15:22
  • 3
    This is really not a good idea. – Jakuje Jul 28 '16 at 21:21
  • 1
    why ?? any alternative ? – M-Jack Jul 29 '16 at 8:14
  • 8
    The PAM is used for other things around the account management in modern systems. Rather than turning it off, you would better be with investigating what is going on in the PAM stack and why does it take so long. – Jakuje Jul 29 '16 at 8:16
  • Leaving very commonly unused PAM module enabled for SSH access is a security hole. Limiting access to critical services such as SSH from security standpoint is always a good idea for ANY other service too. When you want PAM module to cooperate with SSH? For example: when you need to integrate it with active directory via winbind, when you need two factor auth with google tokens, etc. In other cases (when using passwd and shadow) shutting it off is perfectly safe. Every user of PAM shall see this: cve.mitre.org/cgi-bin/cvekey.cgi?keyword=pam – Michal Sokolowski Jun 15 at 7:05

This happened on two of my Fedora 25 servers, and was due to lots of failed SSH login attempts.

(The common suggestions of using GSSAPIAuthentication=no and UseDNS=no, or restarting systemd-logind, made no difference.)

On these servers, /etc/pam.d/postlogin contains:

session     optional      pam_lastlog.so silent noupdate showfailed

The man page for pam_lastlog explains that the showfailed option will:

Display number of failed login attempts and the date of the last failed attempt from btmp.

On these servers, the /var/log/btmp files were enormous due to many failed login attempts. The btmp log files weren't being rotated either.

I installed the logrotate package to ensure the log files will be rotated in future. (On Fedora, the configuration shipped with logrotate itself handles the rotation of /var/log/btmp.)

I also deleted the enormous btmp log files; as soon as I did this, connecting to the servers was instantaneous again.

  • This solved my problem! Thank you. Nice catch. SSH was taking 5-10 seconds, and now it's less than a blink of an eye. This is on a VM that I've had connected to the public Internet for years. Its firewall rules could probably be tuned slightly better, now that I think of it. To others, this is all I did: sudo truncate -s 0 /var/log/btmp - Mine was 2.7G in size. – Carl Bennett Feb 2 at 0:50

For me this issue is caused by large (hundreds of MBs) btmp file. This file logs login attempts. When people are trying to brute force your password this file can be big and cause delays in the "pledge: network" phase.

Try to clear log file

echo "" > /var/log/btmp

and see if it helps.

  • 3
    This needs a lot more explanation. For starters, why do you think this is helpful? – Sven Jun 15 '17 at 11:05

I noticed the following line in my debug feedback:

Control socket connect(/var/lib/jenkins/.ssh/USER@HOST:22): Permission denied

Which was a file that was owned by root:root while I'm jenkins. Removing this file resolved my issues.

In my case the reason was a crashed rsyslogd. I found this out because there were no more log-messages in e.g. /var/log/syslog or /var/log/mail.log

So service rsyslog restart resolved the problem for us.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.