84

Connection to one of my servers using ssh takes more than 20 seconds to initiate.

This is not related to LAN or WAN conditions, since connection to itself takes the same (ssh localhost). After connection is finally establised, it is super fast to interract with the server.

Using -vvv shows that the connection is stuck after saying "pledge: network". At this point, authentication (here using key) is already done, as visible here :

...
debug1: Authentication succeeded (publickey).
Authenticated to myserver.mydomain.com ([xx.xx.xx.xx]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network

(...stuck here for 15 to 25 seconds...)

debug1: client_input_global_request: rtype [email protected] want_reply 0
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
...

Server is Ubuntu 16.04. It already happened to me in the past with another server (was Ubuntu 12.04) , nerver found the solution and the problem disapeared after a while...

sshd_config is the default one provided by Ubuntu.

So far I have tried :

  • using -o GSSAPIAuthentication=no in the ssh command
  • using password instead of a key
  • using UsePrivilegeSeparation no instead of yes, in sshd_config
Improve this question
6
  • 1
    Usually for me slow SSH connections are DNS problems, might that be the case here? For example, the server may be stuck trying to do a reverse DNS for the client's IP and waiting for that to time out
    – Eric Renouf
    Jul 28, 2016 at 14:09
  • 1
    Actually no : by default UseDNS is not defined in sshd_config and man page says that this option is "no" by default.
    – M-Jack
    Jul 28, 2016 at 15:17
  • 4
    Some Googling suggests this can be caused by updating systemd without rebooting. And there was a systemd update for xenial on July 12. systemctl restart systemd-logind fixes the problem only for a short period of time for me.
    – Ivan Kozik
    Aug 15, 2016 at 18:19
  • Or if you're seeing pam_systemd(sshd:session): Failed to create session: Connection timed out as mentioned in an answer, this might be github.com/systemd/systemd/issues/2925
    – Ivan Kozik
    Aug 15, 2016 at 19:17
  • I came here having had this problem after an update, and @IvanKozik's suggestion fixed the problem - i.e systemctl restart systemd-logind - so thanks for that.
    – Paul M
    Nov 23, 2016 at 23:45

14 Answers 14

Reset to default
62

This is probably an issue with D-Bus and systemd. If the dbus service is restarted for some reason, you will also need to restart systemd-logind.

You can check if this is the issue by opening the ssh daemon log (on Ubuntu it should be /var/log/auth.log) and check if it has these lines:

sshd[2721]: pam_systemd(sshd:session): Failed to create session: Connection timed out

If yes, just restart systemd-logind service:

systemctl restart systemd-logind

I had this same issue on CentOS 7, because the messagebus was restarted (which is how the D-Bus service is called on CentOS).

Improve this answer
2
  • I tried to restart systemd-logind but after a while it says PolicyKit daemon disconnected from the bus. We are no longer a registered authentication agent. Job for systemd-logind.service failed because a timeout was exceeded. See "systemctl status systemd-logind.service" and "journalctl -xe" for details.
    – Kun Ren
    May 12, 2017 at 3:49
  • @KunRen you probably need to restart the polkit service using systemctl restart polkit.
    – Strahinja Kustudic
    Jun 27, 2017 at 18:57
34

found the answer :

changed UsePAM from yes to no in sshd_config file

After restarting the ssh service, the connection is now immediate to the server. On this server, PAM is linked to ldap, so that is probably the reason, even if here I am connecting with a user declared on the server itself, not an LDAP one.

Well, this is more a way to bypass the issue, not really a solution... I have other servers set up the same way that are not having this issue.

Hope this may help someone...

Improve this answer
6
  • 2
    changing UsePAM to no has other effects. See this discussion So I had to set a password to the user, because I got errors like User nagios not allowed because account is locked
    – M-Jack
    Jul 28, 2016 at 15:22
  • 7
    This is really not a good idea.
    – Jakuje
    Jul 28, 2016 at 21:21
  • 2
    why ?? any alternative ?
    – M-Jack
    Jul 29, 2016 at 8:14
  • 11
    The PAM is used for other things around the account management in modern systems. Rather than turning it off, you would better be with investigating what is going on in the PAM stack and why does it take so long.
    – Jakuje
    Jul 29, 2016 at 8:16
  • 3
    Leaving very commonly unused PAM module enabled for SSH access is a security hole. Limiting access to critical services such as SSH from security standpoint is always a good idea for ANY other service too. When you want PAM module to cooperate with SSH? For example: when you need to integrate it with active directory via winbind, when you need two factor auth with google tokens, etc. In other cases (when using passwd and shadow) shutting it off is perfectly safe. Every user of PAM shall see this: cve.mitre.org/cgi-bin/cvekey.cgi?keyword=pam
    – Michal Sokolowski
    Jun 15, 2018 at 7:05
22

This happened on two of my Fedora 25 servers, and was due to lots of failed SSH login attempts.

(The common suggestions of using GSSAPIAuthentication=no and UseDNS=no, or restarting systemd-logind, made no difference.)

On these servers, /etc/pam.d/postlogin contains:

session     optional      pam_lastlog.so silent noupdate showfailed

The man page for pam_lastlog explains that the showfailed option will:

Display number of failed login attempts and the date of the last failed attempt from btmp.

On these servers, the /var/log/btmp files were enormous due to many failed login attempts. The btmp log files weren't being rotated either.

I installed the logrotate package to ensure the log files will be rotated in future. (On Fedora, the configuration that ships with logrotate handles the rotation of /var/log/btmp.)

I also deleted the enormous btmp log files; as soon as I did this, connecting to the servers was instantaneous again.

Improve this answer
4
  • 2
    This solved my problem! Thank you. Nice catch. SSH was taking 5-10 seconds, and now it's less than a blink of an eye. This is on a VM that I've had connected to the public Internet for years. Its firewall rules could probably be tuned slightly better, now that I think of it. To others, this is all I did: sudo truncate -s 0 /var/log/btmp - Mine was 2.7G in size.
    – Carl Bennett
    Feb 2, 2018 at 0:50
  • 1
    This was the issue for me, thanks @CarlBennett
    – diegoperini
    Apr 21, 2020 at 12:52
  • 1
    This fixed it for me after a long frustrating search. Thank you!
    – Theron S
    Jul 7, 2020 at 19:14
  • This answer solved my problem. It appears this is a known PAM bug: github.com/linux-pam/linux-pam/issues/270. The "showedfail" option enabled by default by many Linux distributions, prints "NNNN failed logins since the last successful login" on every log in, but to calculate this NNNN it needs to go over the entire /var/log/btmp file, which just grows and grows and after a few years, can become enormous and take over a minute to process during each login!
    – Nadav Har'El
    Aug 22, 2021 at 14:57
19

On Ubuntu 16+ every time I have seen ssh -v XXX@YYY stalling at pledge: network it can be fixed by following the instructions I found here A comprehensive guide to fixing slow SSH logins. Specifically, an optional PAM module that does not appear to be needed is causing the delay.

In /etc/pam.d/common-session on the machine you see slow logins for (ie. the server). Comment out the line session optional pam_systemd.so. That should immediately fix the problem.

This avoids having to completely shut down PAM which cripples login with passwords.

Improve this answer
8
  • 1
    This works in my case, and the link is awesome too. Many thanks!
    – Arnie97
    Jun 29, 2020 at 19:34
  • This seems to work on a Raspberry but I would like to know what it actually does.
    – bomben
    Dec 29, 2020 at 14:55
  • It works! On CentOS8, the corresponding file is /etc/pam.d/password-auth
    – haolee
    Aug 26, 2021 at 13:31
  • Thanks, the page you linked to is very helpful and informative!
    – Pierre
    Apr 6, 2022 at 10:56
  • The system I'm having this trouble on is CentOS 7. That file /etc/pam.d/common-session does not exist, and none of the other solutions I have seen are working. The server is speaking to dirsrv@389 for ldap authentication, but so are a number of other servers that do not take 11 seconds to log in with an ssh key.
    – elyograg
    Dec 13, 2022 at 4:16
7

The problem for me (Ubuntu 19.10) was that my:

/etc/pam.d/sshd

# Print the message of the day upon successful login.
# This includes a dynamically generated part from /run/motd.dynamic
# and a static (admin-editable) part from /etc/motd.
session    optional     pam_motd.so  motd=/run/motd.dynamic
session    optional     pam_motd.so noupdate

Commenting the motd settings got me right in.

Improve this answer
3
  • 1
    Thanks, this was my problem. This takes a toll on a Raspberry Pi 2.
    – Giovanni Bassi
    Nov 1, 2020 at 18:34
  • 1
    This fixes my issues on a Raspberry Pi, probably caused by slow microSD speeds.
    – rtaft
    Nov 11, 2020 at 19:32
  • This also fixed the slow login for SSHD service running inside 20.04 Ubuntu Subsystem for Windows.
    – Dale C. Anderson
    Jul 7, 2021 at 23:48
5

For me this issue is caused by large (hundreds of MBs) btmp file. This file logs login attempts. When people are trying to brute force your password this file can be big and cause delays in the "pledge: network" phase.

Try to clear log file

echo "" > /var/log/btmp

and see if it helps.

Improve this answer
3
  • 4
    This needs a lot more explanation. For starters, why do you think this is helpful?
    – Sven
    Jun 15, 2017 at 11:05
  • tip: Just typing :> /var/log/btmp does the same btw.
    – Marius
    Mar 7, 2019 at 18:37
  • 1
    Sven, a few years after Marek's answer it is now listed as a PAM bug: github.com/linux-pam/linux-pam/issues/270. The "showedfail" option enabled by default by many Linux distributions, prints "NNNN failed logins since the last successful login" on every log in. But to calculate this NNNN it needs to go over the entire /var/log/btmp file, which just grows and grows and after a few years, can become enormous and take over a minute to process during each login! Sad, but true. I had the same problem and removing /var/log/btmp fixed it.
    – Nadav Har'El
    Aug 22, 2021 at 14:59
3

For me the first clue was provided by

UseDNS no

to the /etc/ssh/sshd_config and then of course service ssh restart (on our Debian/Jessie server).

before:

ssh git@git.*****.de true  0.03s user 0.01s system 0% cpu 13.440 total
ssh git@git.*****.de true  0.03s user 0.01s system 0% cpu 20.990 total
ssh git@git.*****.de true  0.03s user 0.02s system 0% cpu 31.114 total
ssh git@git.*****.de true  0.03s user 0.01s system 0% cpu 25.898 total

after:

ssh git@git.*****.de true  0.03s user 0.02s system 5% cpu 0.832 total
ssh git@git.*****.de true  0.03s user 0.01s system 7% cpu 0.523 total
ssh git@git.*****.de true  0.03s user 0.01s system 7% cpu 0.574 total

This revealed that my DNS configuration was wrong (I had a typo in the DNS address). After fixing the IP and restoring the setting UseDNS yes everything worked fine.

Improve this answer
4
  • 1
    No, adding UseDNS no is a solution for a completely different problem.
    – kasperd
    Oct 30, 2018 at 8:57
  • 1
    @kasperd It doesn't matter. In my case I had the very same symptoms (briefly: stuck after saying "pledge: network") and this is what finally helped, so this is a solution to at least a very similar problem and I am sure it will help one or the other at some point.
    – tamasgal
    Oct 30, 2018 at 12:12
  • Same here, two hangs during connection, one after sign_and_send_pubkey, a longer one after pledge: network. Adding only UseDNS no with subsequent service ssh restart did resolve the problem on an old Ubuntu 14.04.5 LTS installation here.
    – Hound
    Mar 26, 2019 at 10:01
  • I forgot to follow up: it was eventually a typo in the DNS address in my configuration. Correcting that and setting UseDNS yes again fixed. So @kasperd was of course right! The root of the problem was deeper and the fix was not targeting the main issue.
    – tamasgal
    Jul 12, 2020 at 9:43
2

In my case the reason was a crashed rsyslogd. I found this out because there were no more log-messages in e.g. /var/log/syslog or /var/log/mail.log

So service rsyslog restart resolved the problem for us.

Improve this answer
1
  • Same cause on a server of ours running CentOS 6.10. Restart of rsyslog took care of it. The thing is, it wasn't dead. It was running, but apparently doing nothing useful.
    – UtahJarhead
    Oct 24, 2018 at 19:02
1

In my case, it is because there are too many logs. You can test if you are in the case, by issuing this command:

sudo journalctl --list-boots

If it takes a while to give results, and give many lines of the result, then, you are in.

To truncate the logs, do this:

sudo journalctl --vacuum-time 2d

It will delete logs which are older than two days.

Improve this answer
1
1

In my case it was a firewall problem after upgrade to debian 11.

Solved by adding:

iptables -t filter -A INPUT -i lo -j ACCEPT

At the beggining of the firewall script.

Improve this answer
0

I noticed the following line in my debug feedback:

Control socket connect(/var/lib/jenkins/.ssh/USER@HOST:22): Permission denied

Which was a file that was owned by root:root while I'm jenkins. Removing this file resolved my issues.

Improve this answer
0

In my case the reason was a crashed rsyslogd. I found this out because there were no more log entries in /var/log/secure

So I Restarted service rsyslog restart resolved the problem for us.

Improve this answer
1
0

I got the same issue and we had configured the SSSD service for AD login.

Stopping the SSSD service fixed the issue for me.

Improve this answer
0

If you are using NIS/YP, make sure nscd is running. In my case, I was getting the following when ssh in.

systemd-logind[30061]: do_ypcall: clnt_call: RPC: Unable to send; errno = Operation not permitted

According to https://github.com/systemd/systemd/issues/7074, nscd should be running to ensure smooth operation.

After I started nscd (which was not running since I upgraded Ubuntu) incoming ssh sessions are much faster.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .