10

I use a perl script (with the Semaphore package) to fire off long-running ssh commands to AWS instances. For several reasons, I do NOT run the ssh commands in the background.

Recently, the Comcast fiber to our office building was cut by a construction crew. We maintain a backup CenturyLink connection, and our IT people switched our office connection over to CenturyLink.

My ssh processes died with a "timeout, server not responding" message when we switched from Comcast to CenturyLink. They died again when we switched back to Comcast after the fiber was repaired.

Is this expected behavior for an open ssh command if the local public IP address changes? If I put the ssh commands in the background, would it solve this issue?

21

Yes, that's expected behavior. Your connection was physically interrupted, after all.

If an interruption in your connection may cause a failure of a long running remote process, consider running it in a tmux session.

| improve this answer | |
  • 2
    @MorganBrown You would use tmux if you want to reconnect to the session. You can't do that with nohup. – Michael Hampton Jul 14 at 14:35
  • 4
    Tmux is really very handy. It is well worth the time to learn it, particularly if you're running long-running jobs over ssh. I highly recommend it. – chicks Jul 15 at 2:36
  • 1
    An IP address change is not a physical interruption. A physical interruption is unplugging a cable. IP addresses are not physical; physics does not apply to them. – user253751 Jul 15 at 8:20
  • 10
    @user253751 The OP suffered a fiber cut and began using their backup ISP. I would consider that a physical interruption. – Michael Hampton Jul 15 at 11:46
  • 1
    tmux or screen. – Gnudiff Jul 15 at 20:46
14

If you do not want to run ssh commands in the background (manually) you may give mosh (mobile shell) a try. It's available for probably any distribution and system including various BSD, Android, Windows and macOS.

| improve this answer | |
  • I thought mosh was more for dealing with irritating latency. Does it handle losing the connection too? – chicks Jul 15 at 2:37
  • 7
    @chicks yes, mosh ("MObile SHell") gracefully handles connection drops and even client IP changes (eg if the client is a mobile device switching cell towers). It makes use of UDP in order to do that. – vikarjramun Jul 15 at 4:39
  • The traditional solution is also to run screen on the server. If your connection is lost, log back in and reconnect to the same screen by running screen -r. – user253751 Jul 15 at 13:07
3

Yes, it's normal.

A TCP server identifies a connection by the combination of local address, local port, remote address and remote port.

Since you talk about "the local public IP address changes" I presume your workplace uses some form of NAT to map the private addresses on your LAN to one or more public addresses on the current internet connection.

Depending on how exactly the NAT is implemented there are several possibilities for what exactly might happen at the packet level, but for regular TCP all of them end up with the TCP connection failing.

There is an extension called "multipath TCP" which (among other things) allows connections to be maintained across network changes, but afaict using it between linux boxes still requires third party kernel modules at this time.

| improve this answer | |
  • 2
    Yes indeed. Another good reason for IPv6 here. If the company had been using an assigned IP block of their own and had a routing agreement with two ISPs, the switchover would have been transparent and no IP addresses would have changed. – Zan Lynx Jul 16 at 17:08
  • 1
    That is really an advantage of going with provider-independent space, provider-independent space generally comes at a cost both to the company using the space (RIR fees, cheaper internet connections usually don't allow it) and to the internet as a whole (routing table size). Granted the costs of PI space are a bit higher with v4 than v6 because you have to actually purchase the addresses, but I suspect that is typically a minor factor. – Peter Green Jul 17 at 16:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.