30

It's a sort of regular occurrence that, after SSHing to a server (or even opening a terminal on my Mac), the login banner prints immediately but it takes ~10 seconds to a minute for the shell prompt to appear. After that, performance is fine and network latency is not unusual.

This doesn't seem like a computationally difficult, memory-intensive, or IO-heavy task. What's it doing with all those billions of CPU cycles?

  • 9
    ssh -v -v -v and ur_shell -x might be prudent debugging steps. – thrig Sep 15 '15 at 17:26
  • 2
    Is there a lot of lines in your .bash_history? – kasperd Sep 15 '15 at 17:37
  • 2
    Look at your .profile and related files (sorry, not sure which shell you're using) and remove temporarily to see if that improves things. You probably have a command(s) that are timing out. – TheFiddlerWins Sep 15 '15 at 17:45
  • 3
    Alternatively to Moby Disk's suggestion, is it still equally slow if you start the shell itself from within the shell? If it's slow when (re)connecting but fast when starting a second shell from within an existing, connected session, that will tell you that it's not the shell itself that is causing the slowness; if it is equally slow in the two situations, then something the shell does on startup is taking a lot of time. Either way you learn some about which aspect of "from zero to shell prompt" is slow. – a CVn Sep 15 '15 at 20:55
  • 2
    Often this is GSSAPI authentication trying to occur (which, if you're not a Kerberos shop, is likely completely useless). Other times, it's DNS reverse lookups. – Charles Duffy Sep 15 '15 at 22:32
33

Quite a few things could be happening here. You can find most of the answers in your shell's manual, but those are usually incredibly long and oblique, so...

Chances are your problem boils down to one of a few things.

If your profile or bashrc have expensive things, consider trimming them back.

If your profile or bashrc use a reverse DNS lookup (to set the prompt or something), fix DNS or use hostname instead.

Shells open a lot of files, among other things, while initialising. If system load is high, it will often show up here.

If the banner is pre-authentication, it might also actually be authentication (pam, LDAP, etc.) that is slow.

It might be none of these things, though. A surprising amount of stuff happens right before displaying the prompt!

| improve this answer | |
  • 1
    Another one I've seen block for a substantial while is attempted GSSAPI authentication (if not turned off on either the SSH client or server config). Using a full-system tracing tool such as sysdig is probably the best silver bullet available to get to the bottom of this kind of cross-concern issue. – Charles Duffy Sep 15 '15 at 22:33
  • +1 Thorough answer. This site has a nice flowchart for logins, which files they execute/source, and other info. – Tim S. Sep 15 '15 at 23:22
  • 15
    +1. 99% of the time for me it's the reverse DNS lookup. – mpontillo Sep 15 '15 at 23:38
  • @Mike: same here - I would recommend to start with this one as it is easy to fix. It was 100% in my case. – WoJ Sep 16 '15 at 15:19
22

It's probably either waiting for DNS or trying to authenticate via LDAP or such.

Try to add UseDNS no to /etc/ssh/sshd_config

If it also does it on local logons, check if any LDAP servers or DNS servers you have configured are slow or unresponsive.

| improve this answer | |
6

One possibility (covered by other answers) is that the process of setting up the SSH session itself is where the time is lost.

Another alternative is that your shell startup scripts running on the remote machine after SSH session establishment have something that is taking a long time (perhaps attempting to access some broken network mount). You can debug this second possibility as follows:

Temporarily add the following to the top of your ~/.bash_profile:

set -x
PS4='+ $(date "+%s.%N")\011 '

The set -x turns on some debugging for every shell command executed. The PS4 variable controls how that debugging is presented - specifically in this case we use date to add timestamps.

You may then analyze the timestamps of the debugging output to see which commands in your startup scripts are taking too long.

| improve this answer | |
  • 1
    True if and only if the issue happens after the remote session is opened. Many of the potential causes are during SSH handshake and authentication. – Charles Duffy Sep 15 '15 at 22:32
  • 2
    Much improved. :) – Charles Duffy Sep 15 '15 at 23:09
3

If it's an Ubuntu server, the default login setup checks if any packages are updatable every time a login shell runs. If the package lists aren't in disk cache, this can take a second or two even on a fast idle desktop.

$ ssh localhost 
Welcome to Ubuntu 15.04 (GNU/Linux 3.19.0-26-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

*** System restart required ***
Last login: Sat Sep 12 01:38:38 2015 from localhost

To generate that "restart required" message, it had to check that the currently running kernel is not the default kernel currently installed. (i.e. there was a kernel and I didn't reboot yet.) It will also print a count of security updates available, if there are any.

I think this is the major slowdown in logging in to Ubuntu that's been introduced recently.

If not that, then your ~/.bash_profile / ~/.bashrc might be the problem.

Have you tried logging in to the server from itself (ssh localhost)? Or logging in a 2nd time right away? (To see if it's a lot faster when stuff is cached.)

| improve this answer | |
2

This is in most cases a timeout of a DNS request.

Cause: The server attempts a reverse DNS lookup using the IP address of the client, and does not get a reply. If A connects to B, B tries to convert the IP address of A to a name.

Workaround: Enter the IP-Address and name of the client to the hosts file of the server.

Solution: make all hosts known to the DNS server.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.