12

I use a perl script (with the Semaphore package) to fire off long-running ssh commands to AWS instances. For several reasons, I do NOT run the ssh commands in the background.

Recently, the Comcast fiber to our office building was cut by a construction crew. We maintain a backup CenturyLink connection, and our IT people switched our office connection over to CenturyLink.

My ssh processes died with a "timeout, server not responding" message when we switched from Comcast to CenturyLink. They died again when we switched back to Comcast after the fiber was repaired.

Is this expected behavior for an open ssh command if the local public IP address changes? If I put the ssh commands in the background, would it solve this issue?

3 Answers 3

20

Yes, that's expected behavior. Your connection was physically interrupted, after all.

If an interruption in your connection may cause a failure of a long running remote process, consider running it in a tmux session.

12
  • 2
    @MorganBrown You would use tmux if you want to reconnect to the session. You can't do that with nohup. Jul 14, 2020 at 14:35
  • 4
    Tmux is really very handy. It is well worth the time to learn it, particularly if you're running long-running jobs over ssh. I highly recommend it.
    – chicks
    Jul 15, 2020 at 2:36
  • 1
    An IP address change is not a physical interruption. A physical interruption is unplugging a cable. IP addresses are not physical; physics does not apply to them.
    – user253751
    Jul 15, 2020 at 8:20
  • 10
    @user253751 The OP suffered a fiber cut and began using their backup ISP. I would consider that a physical interruption. Jul 15, 2020 at 11:46
  • 1
    tmux or screen.
    – Gnudiff
    Jul 15, 2020 at 20:46
14

If you do not want to run ssh commands in the background (manually) you may give mosh (mobile shell) a try. It's available for probably any distribution and system including various BSD, Android, Windows and macOS.

3
  • I thought mosh was more for dealing with irritating latency. Does it handle losing the connection too?
    – chicks
    Jul 15, 2020 at 2:37
  • 7
    @chicks yes, mosh ("MObile SHell") gracefully handles connection drops and even client IP changes (eg if the client is a mobile device switching cell towers). It makes use of UDP in order to do that. Jul 15, 2020 at 4:39
  • 1
    The traditional solution is also to run screen on the server. If your connection is lost, log back in and reconnect to the same screen by running screen -r.
    – user253751
    Jul 15, 2020 at 13:07
3

Yes, it's normal.

A TCP server identifies a connection by the combination of local address, local port, remote address and remote port.

Since you talk about "the local public IP address changes" I presume your workplace uses some form of NAT to map the private addresses on your LAN to one or more public addresses on the current internet connection.

Depending on how exactly the NAT is implemented there are several possibilities for what exactly might happen at the packet level, but for regular TCP all of them end up with the TCP connection failing.

There is an extension called "multipath TCP" which (among other things) allows connections to be maintained across network changes, but afaict using it between linux boxes still requires third party kernel modules at this time.

2
  • 2
    Yes indeed. Another good reason for IPv6 here. If the company had been using an assigned IP block of their own and had a routing agreement with two ISPs, the switchover would have been transparent and no IP addresses would have changed.
    – Zan Lynx
    Jul 16, 2020 at 17:08
  • 1
    That is really an advantage of going with provider-independent space, provider-independent space generally comes at a cost both to the company using the space (RIR fees, cheaper internet connections usually don't allow it) and to the internet as a whole (routing table size). Granted the costs of PI space are a bit higher with v4 than v6 because you have to actually purchase the addresses, but I suspect that is typically a minor factor. Jul 17, 2020 at 16:12

Not the answer you're looking for? Browse other questions tagged .