Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Mac OS X server (referred to as the internal server behind a firewall that is reverse port forwarding via ssh -gnN -R *:remoteport:localhost:localport root@remote.server where remote.server is the domain/IP of a public server, which I will call the remote server. Then, I can connect to the internal server by connecting toremote.server:remoteport.

Periodically, the tunnel will fail, and to that end I have a script on the internal server that tries to reconnect to the remote server until it succeeds. This has worked on several occasions.

But recently, I experienced an error in which the internal server loses its connection to the remote server, and nothing I do on the internal server can restore the connection. The remote server must be rebooted, after which the tunnel is immediately reestablished by the script.

The remote server is a VPS running a base install of CentOS 6. No configuration has changed in the last few months. The remote server ran the tunnel without errors for over 4 months, then failed twice in the last 12 hours in this manner, requiring two reboots only hours apart.

What kinds of things could cause this behavior, and where should I start looking for problems?

share|improve this question

1 Answer 1

up vote 1 down vote accepted

More than likely the client connection was lost, but the remote server has not yet closed down the connection. This results in a port conflict as the old session will still be listening on the port you're attempting to bind to.

Log in to the remote server, identify the stale sshd process, and send it HUP signal (kill -HUP) to terminate the session and its associated listener.

That said, you really shouldn't be relying on this type of "persistent" tunnel in a production environment. (especially if you're using it to skirt a firewall policy that is in place for a reason)

share|improve this answer
    
That makes sense. Is there a way to make the remote server automatically close the connection when it's lost? –  Justin Mrkva Jan 11 at 2:15
    
Also, it's not really a "production" environment, it's for personal use. The problem is that the provided on-site VPN isn't mobile compatible and so this is the only way to make it work. (I've been given the go ahead to use the reverse tunnel as well). –  Justin Mrkva Jan 11 at 2:17
    
The remote side will eventually terminate the connection when it notices that the connection is stale, but this can take a very long time depending on the nature of the dropped connection. Your best bet is to take a look at man sshd_config and set ClientAliveInterval to a non-zero value on the remote server. –  Andrew B Jan 11 at 16:19
    
I set ClientAliveInterval to 15 and ClientAliveCountMax to 4. That should be 1 minute before restarting. However, it did it again today and did not reset; I had to reboot the remote server again. I also tried running sudo service sshd restart but it didn't help. Only a reboot solved it, so it doesn't look like a rogue sshd process. –  Justin Mrkva Jan 15 at 11:33
    
Found what might be the problem - memory leaks in sshd. Thing is, when I run ps aux or top on the remote server, I can't find any processes using any substantial memory; they all show 0.1% or lower. I'm posting that as a separate question. –  Justin Mrkva Jan 17 at 11:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.