up vote 1 down vote favorite
share [g+] share [fb]

This happened for the first time three days ago - I ssh to the server, authenticate using a password, get the welcome message but it remains hanging on the "Last login:..." line. The command line doesn't show and the server doesn't react to my input. Other services on the server keep working ok (apache, tomcat, database, ..). The box has an out-of-band management using which I was able to restart it. After the restart the ssh worked ok again and I didn't find anything suspicious in the logs. Three days later the same problem occurs on this box again, and newly on yet another server in the cluster - 100% same symptoms.

Both servers have about 2 month old installation of Debian Squeeze (6.0.2) and the problem never occurred before despite frequent ssh-ing, so it should not be a problem of settings. We haven't been installing anything new for quite some time now. I also made sure there is enough disk space on both servers. Since it started to happen all of a sudden on two servers at about the same time, I suspect some bug may have been introduced via Debian updates, yet I haven't been able to find anyone with the same problem.

Most similar issues I have found:

ssh freezes at the "Last Login Line" - in our case everything worked fine until recently, so nothing related to settings should be our problem. Diskspace checked, I couldn't check the memory but I would expect something would be in the logs if the system had been running out of it.

Remote Fedora system unresponsive, odd but consistent behavior when trying to log in - problem with high load on the server; unlike in this case, nothing changes even if I wait for 10+ minutes

link|improve this question
sorry for misplacing (had seen some related questions on SO and didn't think twice). Thank you guys for migrating my question. – Pavel H Oct 3 '11 at 11:40
What about 'ssh -T' or 'ssh myhost bash -i'? Do one or both work? Is it a Dell system? – Matt Apr 27 at 20:59
feedback

migrated from stackoverflow.com Oct 3 '11 at 11:29

This question came from our site for professional and enthusiast programmers.

3 Answers

up vote 2 down vote

Login using ssh -vvv and see what debug info shows up. This is odd in that the last login line is usually after the ssh session is setup. Also check your /var/log/{messages,auth,daemon} and whatever else is relevant.

Are your drives full? Is something unexpected going on with configuration? I know you didn't change it, but maybe something was compromised?

link|improve this answer
Hi Jeff, ssh debug info looks ok (same as when I ssh to a working system), it ends up with debug2: shell request accepted on channel 0, then the welcome lines show and it gets stuck. Nothing in messages.log and daemon.log. auth.log says session opened for user... (and then session closed.. - probably as I closed the terminal on my computer). I also suspect the problem may not be in ssh itself - it seems to have completed its job and the problem happens afterwords. – Pavel H Oct 3 '11 at 12:52
...and I did check the diskspace - no prob there, and there has not been any change in the configuration recently – Pavel H Oct 3 '11 at 12:58
I wager your SSH config is just fine and something is going on in your login scripts -- probably a process that is hanging. – Jeff Ferland Oct 3 '11 at 13:23
makes sense. At the moment I have one of the servers stuck as described - is there anything I could do now to help me identify the process before I restart the server? I tried ctrl-c, ctrl-d and ctrl-z hoping perhaps I could escape but no. I have no physical access to the box. – Pavel H Oct 3 '11 at 16:43
In desperation, I'd attempt ssh -t screen, ssh -t bash, or even ssh -t vim... if one of those works, you can run commands from it. Vim might be the best bet because it won't spawn a shell at the start, so that may avoid login shell issues and you can spawn a non-login shell from it. – Jeff Ferland Oct 3 '11 at 17:05
show 3 more comments
feedback
up vote 0 down vote

If the hang disappears after say 1 minute to two minutes, you probably have an unreachable DNS server problem.

If it is not that, consider using ServerAliveInterval, KeepAlive yes and ClientAliveInterval (e.g. 60)

link|improve this answer
Hi, no, the hang doesn't disappear - waited for more than 10 minutes. Once it happens, the only way to be able to ssh in again is to restart the server via DRAC. – Pavel H Oct 3 '11 at 11:39
feedback
up vote 0 down vote

You have something unexpected in your configuration. Home directories under nfs/samba/something else? Something executing at your .bashrc or other startup files?

link|improve this answer
Hi Janne, nothing I know of. Home directories are local to the servers. I checked /etc/bash.bashrc, /etc/profile (nothing in profile.d) and ~/.bashrc - no fancy stuff in any of them, just setting the environment (prompt, autocomplete etc) - guess it is just default stuff. Is there anything else I could check? – Pavel H Oct 3 '11 at 16:49
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.