Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This happened for the first time three days ago - I ssh to the server, authenticate using a password, get the welcome message but it remains hanging on the "Last login:..." line. The command line doesn't show and the server doesn't react to my input. Other services on the server keep working ok (apache, tomcat, database, ..). The box has an out-of-band management using which I was able to restart it. After the restart the ssh worked ok again and I didn't find anything suspicious in the logs. Three days later the same problem occurs on this box again, and newly on yet another server in the cluster - 100% same symptoms.

Both servers have about 2 month old installation of Debian Squeeze (6.0.2) and the problem never occurred before despite frequent ssh-ing, so it should not be a problem of settings. We haven't been installing anything new for quite some time now. I also made sure there is enough disk space on both servers. Since it started to happen all of a sudden on two servers at about the same time, I suspect some bug may have been introduced via Debian updates, yet I haven't been able to find anyone with the same problem.

Most similar issues I have found:

ssh freezes at the "Last Login Line" - in our case everything worked fine until recently, so nothing related to settings should be our problem. Diskspace checked, I couldn't check the memory but I would expect something would be in the logs if the system had been running out of it.

Remote Fedora system unresponsive, odd but consistent behavior when trying to log in - problem with high load on the server; unlike in this case, nothing changes even if I wait for 10+ minutes

share|improve this question

migrated from stackoverflow.com Oct 3 '11 at 11:29

This question came from our site for professional and enthusiast programmers.

    
sorry for misplacing (had seen some related questions on SO and didn't think twice). Thank you guys for migrating my question. –  Pavel H Oct 3 '11 at 11:40
    
What about 'ssh -T' or 'ssh myhost bash -i'? Do one or both work? Is it a Dell system? –  user119058 Apr 27 '12 at 20:59
add comment

3 Answers 3

Login using ssh -vvv and see what debug info shows up. This is odd in that the last login line is usually after the ssh session is setup. Also check your /var/log/{messages,auth,daemon} and whatever else is relevant.

Are your drives full? Is something unexpected going on with configuration? I know you didn't change it, but maybe something was compromised?

share|improve this answer
    
Hi Jeff, ssh debug info looks ok (same as when I ssh to a working system), it ends up with debug2: shell request accepted on channel 0, then the welcome lines show and it gets stuck. Nothing in messages.log and daemon.log. auth.log says session opened for user... (and then session closed.. - probably as I closed the terminal on my computer). I also suspect the problem may not be in ssh itself - it seems to have completed its job and the problem happens afterwords. –  Pavel H Oct 3 '11 at 12:52
    
...and I did check the diskspace - no prob there, and there has not been any change in the configuration recently –  Pavel H Oct 3 '11 at 12:58
    
I wager your SSH config is just fine and something is going on in your login scripts -- probably a process that is hanging. –  Jeff Ferland Oct 3 '11 at 13:23
    
makes sense. At the moment I have one of the servers stuck as described - is there anything I could do now to help me identify the process before I restart the server? I tried ctrl-c, ctrl-d and ctrl-z hoping perhaps I could escape but no. I have no physical access to the box. –  Pavel H Oct 3 '11 at 16:43
    
In desperation, I'd attempt ssh -t screen, ssh -t bash, or even ssh -t vim... if one of those works, you can run commands from it. Vim might be the best bet because it won't spawn a shell at the start, so that may avoid login shell issues and you can spawn a non-login shell from it. –  Jeff Ferland Oct 3 '11 at 17:05
show 3 more comments

If the hang disappears after say 1 minute to two minutes, you probably have an unreachable DNS server problem.

If it is not that, consider using ServerAliveInterval, KeepAlive yes and ClientAliveInterval (e.g. 60)

share|improve this answer
    
Hi, no, the hang doesn't disappear - waited for more than 10 minutes. Once it happens, the only way to be able to ssh in again is to restart the server via DRAC. –  Pavel H Oct 3 '11 at 11:39
add comment

You have something unexpected in your configuration. Home directories under nfs/samba/something else? Something executing at your .bashrc or other startup files?

share|improve this answer
    
Hi Janne, nothing I know of. Home directories are local to the servers. I checked /etc/bash.bashrc, /etc/profile (nothing in profile.d) and ~/.bashrc - no fancy stuff in any of them, just setting the environment (prompt, autocomplete etc) - guess it is just default stuff. Is there anything else I could check? –  Pavel H Oct 3 '11 at 16:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.