11
  • Client : OpenSSH_5.1p1 Debian-5ubuntu1 (Ubuntu 9.04)
  • Server : OpenSSH_5.1p1 Debian-5 (Proxmox 2.6.24-7-pve)

I use SSH to execute commands remotely on the server (module check_by_ssh of Nagios). But SSH hangs from time to time when trying to execute commands. I can log to the server via SSH but not executing a simple 'ls'. And it seems to block from all clients from the same IP address. Authentication is not the problem, may it be made by SSH keys or password.

ssh -l root -p 2222 server.domain.tld 'ls'

Here the client debug info

debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env ORBIT_SOCKETDIR
*** skipping approx 40 env var ignored
debug1: Sending command: ls
debug2: channel 0: request exec confirm 1

It hangs there. Then after a random time, it works again (without doing anything). Killing all sshd process on the server seems to work too. It works from a Putty. I saw that some people had trouble like this due to ISP reverse DNS problem, but it does not seem to be the case here.

It can work for hours and then not work for half an hour or so.

What could explain this behaviour ?

EDIT : Seems that with -t or -T option, ssh does not hang, but I can't pass one of these options in the check_by_ssh of nagios

1
  • Our two Proxmox hypervisor do the same thing randomly, quite allways at the same time. They are 64 bits architecture hosted remotely by OVH but no it the same bay it seems.
    – Serty Oan
    Jun 16, 2010 at 7:52

12 Answers 12

11

I had the same problem, and today finally discovered what was causing the issue (for me at least). This might help you too.

When ssh is setting up a session, the DSCP flags field in the IP header is set to 0x0. If you establish an interactive session, it is set to 0x10 (16), and if you establish a non-interactive session, it is set to 0x8 (8). The ssh client sets the DSCP field with the setsockopt() system call (which I verified in the source)

A faulty configuration on a VPN at my employer was dropping the packets with the DSCP of 0x8, causing all non-interactive ssh traffic to also get dropped. To verify it was the DSCP field that was causing the drop, I used iptables on the ssh server to force the DSCP field to be set to 0x16 and tested my non-interactive traffic (ssh ls, same thing you were trying) and it worked after that. You might also try the same thing and see if thats why your session is hanging.

To set DSCP to 0x10 on all outgoing ssh traffic from your ssh server, run:

$ sudo iptables -t mangle -A OUTPUT -p tcp --sport 22 -j DSCP --set-dscp 0x19

This was on a rhel 6.5 box.

2
  • 3
    Holy cow, this is the craziest low-level solution I've seen to an SSH problem, never would've expected the problem to be this low in the stack. May 15, 2018 at 3:31
  • 1
    It's also possible to specify the DCSP flags that SSH uses for interactive and noninteractive sessions using the IPQoS option in the ssh or sshd config files. May 25, 2022 at 14:03
7

I got idea to resolve my problem from this blog. I also have very interesting problem

I got a L2vpn link (vendor provided MPLS L2) to connect my HO and branch office. all ping connectivity testings were working fine. When i ssh using debian server from HO to a debian server at client side i can log in to that server but after remotely ssh login to branch server i was unable to run ifconfig, htop or ps -ef commands. When i apply thoses commands the session freezes. Evn that i check it from windows pc using putty the result was same. Interesting thing is that when i use putty manager and ssh via that application from win 7 pc it was working fine. After reading this blog i got mpls mtu information from service provider and try the same scenario with different mtu size on source debian server interface at HO. Finally mtu sizes from 1440 to 1470 was working fine where as defaults mtu size 1500 was not working. Conclusion: the both end debian server's mtu size was default ie 1500 but in the mid way where service providers MPLS L2vpn mtu size was miss matching. thanks

2
3

You may be hitting an SSH rate limiter on the server-side network. This is a firewall technique to block IP addresses that have too many new connection requests within a short period of time. Then the source IP is blocked for a defined period of time.

3

Possibly an ICMP Path MTU Discovery issue.

In our cause, all ICMP Paramters were blocked by the firewall on the server side. Decreasing client side MTU (adviced by this text) solved the issue temporary. But after allowing all (but redirect) ICMP parameters at the server side, the problem gone.

1

I've experienced the same when having MTU problems. Using ciscos ipsec client-to-site, and then openvpn on top of that. Basically any packet with the size of 1500 bytes would freeze the session.

2
  • And did you find a solution ?
    – Serty Oan
    Oct 5, 2010 at 9:12
  • Set a lower MTU in the Cisco VPN client :-)
    – 3molo
    Oct 5, 2010 at 9:14
1

On my linux machine the VMWare network adapter vmnet2 MTU was set to 1500. The network is used for a virtual machine acting as internet gateway. Since changing the source from Wifi Internet to PPPoE Internet (fiber optics), remote ssh was not working anymore. After lowering vmnet2'S mtu to 1400 ssh remote commands to other ssh servers worked again.

ifconfig vmnet2 mtu 1400 up
1

I used the -tt flag with my remote ssh command and it fixed my issue with the client hanging after running a command remotely on a server

E.g. ssh -tt [email protected] "echo 'command that causes hang here'"

1
  • I think this application level flag (forcing TYY allocation) might also set the lower level DSCP 'interactive' bit described in @Unacoder 's answer!
    – DouglasDD
    Aug 19, 2023 at 19:58
0

Check the ssh on the server side. You may "strace" the created process / mail sshd process and see what syscalls is it calling. This should give you more info on what is it doing.

Also try "touch /tmp/randomfile" and see if the hangs happens after it get created or afterwards.

1
  • What should I look for in the strace ? Does not look different when I just log via ssh or execute a command touch /tmp/randomfile hangs but no file is created
    – Serty Oan
    Jun 2, 2010 at 12:04
0

Have you checked to make sure there arent any PAM errors? just because it works from putty doesnt mean authentication isnt the problem.

9
  • The authentication is not the problem because the debug log of client and server show that authentication has passed.
    – Serty Oan
    May 28, 2010 at 14:38
  • is the directory being listed local to the machine it is running on, and are there any disk/filesystem errors?
    – Devnull
    May 28, 2010 at 19:36
  • (Sorry did not see your answer) It is local to the machine, no disk errors. It hangs whatever command is done, may it be a ls or a touch etc
    – Serty Oan
    Jun 2, 2010 at 12:06
  • has this machine been renamed or readdressed since it was installed?
    – Devnull
    Jun 2, 2010 at 14:33
  • Not renamed nor readdressed. And what's strange is that it makes the same on another proxmox server, but not at all on other servers nor vms on those proxmox.
    – Serty Oan
    Jun 2, 2010 at 14:47
0

I had similar problem. Both client's and server's MTU were 9000. After I lowered client's MTU to 1500 problem was gone.

1
  • Both MTU are at 1500 here, but thanks
    – Serty Oan
    Oct 14, 2010 at 8:22
0

A hang on "sending command" can be caused by SSH actually waiting for the keyphrase/password on the key. One can find out if this is the case by just taking off the command and SSH'ing into the server without a command at the end. It would then ask for a passphrase.

-1

Could there be anything in the user shell environment deadlocking or delaying on open? The -t/-T workaround and closing other ssh sessions clearing it up sounds like something acquiring a lock with the assumption it's the only process.

2
  • Indeed, any idea of options/config that could remove those locks ?
    – Serty Oan
    Sep 6, 2010 at 7:50
  • Look through your shell rc files for anything non-standard. Assuming bash, ~/.bashrc, ~/.bash_profile, and the corresponding /etc/bash* files if it happens for multiple users. Perhaps you can compare with another system (or another user if it's only to one user)?
    – Jeremy M
    Sep 16, 2010 at 1:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .