I have a set of issues, all combining to make for one nasty problem.
The background: I have a server behind a firewall reverse tunneling using an external VPS (we'll call the server "the server" and the VPS that it's connecting to "the tunnel machine"). The SSH tunnel will randomly die. I have a script on the server to reconnect, but unfortunately, the SSHD daemon on the tunnel machine (running CentOS 6.5) will often NOT shut down. It just stays open, which means the reverse tunnel can't bind to the ports.
I've set ClientAliveInterval to 15 and ClientAliveCountMax to 2 on the tunnel machine in order to force the server to quit listening if more than 30 seconds goes by. Sometimes it works... sometimes it doesn't. In many cases, it will just sit there forever listening on ports for a connection that's no longer there, and since it won't give up the ports, ssh can't bind to them.
I also have two secondary problems. When I inevitably get
Write failed: Broken pipe on the ssh command on the server, it does not try to reconnect. SSH just sits there waiting for input. Similarly, the message
Connection to xxx.xxx.xxx.xxx timed out while waiting to write comes up sometimes. On the client, I have ServerAliveInterval at 15 and ServerAliveCountMax at 4, so it waits 1 minute before attempting to reconnect. Still, this isn't enough, since this message can cause SSH to hang indefinitely. For remote port forwarding failure warnings, I use
-o ExitOnForwardFailure=yes to make sure SSH dies if it receives one of these warnings, ensuring my script knows about the failure and can retry. But for
Write failed: Broken pipe and others I can't find any such option, and every time one of these errors comes up, SSH never dies, and it doesn't restart.
First, what's up with the sshd process not terminating when its connection is lost, and second, how do I ensure EVERY disconnect or warning causes SSH to terminate to ensure it can try to reconnect?
Also, yes, before anyone mentions it: I have tried autossh, and it suffers from these exact same shortcomings.