208

Is there a way to connect to an ssh session that was disconnected? We are having problems with our network connection to a remote site that we are working on separately; however, in the mean time we experience a large number of disconnects due to lost packets while connected to servers at the remote location. Many times the session stays active for a while, and sometimes it happens to be in the middle of some action (file editing, running some process, etc...) that I need to get back to rather than restart if possible.

2

15 Answers 15

161

UPDATE: For an actual answer see zero_r's answer below

This isn't an answer, but a workaround. Use screen.

When you first log in, run screen. You get another shell, run commands in that. If you're disconnected, the screen process keeps the terminal alive so that your shell and the processes it is running don't fall over. When you reconnect, run 'screen -r' to resume.

There's a bunch more to configuring and using screen, but the above should workaround your problem.

5
  • 20
    Or use tmux - a 'cooler' screen, I've used both and much prefer tmux because you can split the term nicely.
    – John Hunt
    Sep 19, 2012 at 13:38
  • 3
    I was happy to find screen was already installed on my Ubuntu 12.x so if you not sure just try this command, it did reconnect and resume the do-release-upgrade process: sudo screen -D -r
    – bjm88
    Jan 11, 2016 at 15:56
  • 1
    @bjm88, sudo should be used only if the original screen session was initiated with sudo. Otherwise it will return as if no screen session exists: There is no screen to be detached
    – Noam Manos
    Jan 28, 2018 at 9:26
  • +1 I realize this isn't the actual answer; however, screen -r was exactly what I was looking for. Jul 16, 2019 at 2:02
  • 2
    For an actual answer see @march_happy's reptyr reply or my how-to below. Mar 10, 2022 at 20:49
179

Try to set ClientAliveInterval (e.g. 60) and TCPKeepAlive (yes or no) to appropriate values on the serverside sshd.conf .

This should keep your session alive even if the connection gets lost for minutes.

11
  • 31
    So far, you seem to be the only one actually answering the question instead of suggesting screen as a workaround.
    – kbyrd
    Jun 4, 2009 at 13:20
  • 3
    Good call. We've found this method particullarly useful when traversing cisco ASA/PIX firewalls, which by default love to time out tcp connections. Jun 4, 2009 at 13:21
  • 8
    Maybe this isn't the right place to ask, but is it possible for the poster to switch answers? We should the 'screen' answer (mine's not the best, I like Mike Pountney's, but I got the rep) in with this information.
    – kbyrd
    Jun 4, 2009 at 13:27
  • 1
    Will this work when connected over a Cisco VPN connection, and the VPN connection is lost and re-established?
    – Brent
    Jun 4, 2009 at 14:26
  • for me screen works perfect, i can reconnect to screened task and countiune to work May 14, 2015 at 11:15
76

As mentioned above, GNU Screen is the way to go. It allows you to have a 'screen session' on the remote box that you can run multiple commands in, via multiple 'screen windows'. This will simply detach if your parent SSH connection dies, keeping all the subprocesses running within it alive and well.

man screen is your friend as usual, and the OS package should be called screen if it is not installed by default.

Basics are:

  • Start a screen session (on your remote host):

      $ screen
    
  • Disconnect from your screen session: CTRL-A, d

  • Reconnect to your screen session after logging back in again:

      $ screen -d -r
    
  • Open another screen 'window': CTRL-A, c

  • Cycle through your open screen windows: CTRL-A, space

There is lots of cool stuff you can do with screen. I've been using it for over 10 years, and I'm still finding out new features. It's my favourite Unix utility.

4
  • 1
    Yea, this is a much better answer than mine.
    – kbyrd
    Jun 4, 2009 at 13:20
  • Can I still connect to the disconnected ssh session, if I reboot my pc where screen is running? May 11, 2018 at 7:35
  • 1
    tmux is more powerful
    – Pegasus
    Sep 6, 2019 at 13:28
  • @BarathVutukuri, I believe you mean you're rebooting the client PC. That's fine--that's exactly what this is for. But that's not where you run screen. From serverfault.com/a/198874: "If you're logging into a remote machine, but need to reboot your local machine, then it's no problem as long as you're running screen on the remote machine instead of your local machine."
    – Mathieu K.
    Oct 28, 2020 at 2:27
62

I can't believe no one has mentioned MOSH;

Mosh is a seperate protocol that can hook into the SSH login process, it keeps your session alive after days of disconnection, changing IP, high latency and so on. It is explained on the home page better than I can explain it so I have copied the description below. My experiences and advice are that I use it on my Android mobile, it's a life saver when travelling and SSH'ing. The same is true on my laptop when tethered with mobile on the train for example. I recommend compiling from source to get the latest version, the repo version for me inside Ubuntu has a few annoyances in it which are fixed in the newest version (at the time of writing).

Mosh (mobile shell)

Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.

Mosh is a replacement for SSH. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.

Mosh is free software, available for GNU/Linux, FreeBSD, Solaris, Mac OS X, and Android.

Features from the website:

  • Change IP. Stay connected: Mosh automatically roams as you move between Internet connections. Use Wi-Fi on the train, Ethernet in a hotel, and LTE on a beach: you'll stay logged in. Most network programs lose their connections after roaming, including SSH and Web apps like Gmail. Mosh is different.

  • Makes for sweet dreams: With Mosh, you can put your laptop to sleep and wake it up later, keeping your connection intact. If your Internet connection drops, Mosh will warn you — but the connection resumes when network service comes back.

  • Get rid of network lag: SSH waits for the server's reply before showing you your own typing. That can make for a lousy user interface. Mosh is different: it gives an instant response to typing, deleting, and line editing. It does this adaptively and works even in full-screen programs like emacs and vim. On a bad connection, outstanding predictions are underlined so you won't be misled.

  • No privileged code. No daemon: You don't need to be the superuser to install or run Mosh. The client and server are executables run by an ordinary user and last only for the life of the connection.

  • Same login method: Mosh doesn't listen on network ports or authenticate users. The mosh client logs in to the server via SSH, and users present the same credentials (e.g., password, public key) as before. Then Mosh runs the mosh-server remotely and connects to it over UDP.

  • Runs inside your terminal, but better: Mosh is a command-line program, like ssh. You can use it inside xterm, gnome-terminal, urxvt, Terminal.app, iTerm, emacs, screen, or tmux. But mosh was designed from scratch and supports just one character set: UTF-8. It fixes Unicode bugs in other terminals and in SSH.

  • Control-C works great: Unlike SSH, mosh's UDP-based protocol handles packet loss gracefully, and sets the frame rate based on network conditions. Mosh doesn't fill up network buffers, so Control-C
    always works to halt a runaway process.

5
  • 13
    Mosh is not an ssh wrapper -- it's a different protocol, almost entirely unrelated to ssh (it only uses ssh for login).
    – jch
    Feb 6, 2015 at 3:04
  • +1 Very true, my mistake.
    – Baldrick
    Oct 10, 2015 at 14:03
  • 6
    What about security, man in middle? encryption ? key exchange ? Feb 9, 2016 at 6:06
  • Mosh looks awesome, although sadly doesn't support port forwarding yet.
    – Boann
    Nov 23, 2018 at 13:09
  • 4
    @NasirIqbal brings up a good point. MOSH appears to use a custom designed protocol built on top of UDP and pieces together its own cipher suite instead of using DTLS. No one other than MOSH seems to use this protocol from a cursory google search. Given the potential importance of what SSH sessions protect and SSH being one of the most battle tested protocols on the planet, I have to say this makes me favor plain SSH+Tmux/Screen even though MOSH does appear to have a lot of nice features. I will watch this project with much interest as it does seem really cool.
    – Dev
    Jul 23, 2021 at 18:33
21

autossh watches your connection and if it goes down, it reconnects. It is more reliable than keepalives. If you connect to a screen session, you'll continue right from where you disconnected (see rscreen that comes with autossh)

4
  • 2
    outside of a configurable keep-alive, it doesn't inherently allow you to reconnect to a session (what the OP was looking for). but autossh can be really helpful for apps that require ssh tunnels, port mapping, etc.
    – ives
    Dec 17, 2014 at 0:26
  • yes autossh+screen would be the sweet spot. autossh has rscreen for this.
    – hayalci
    Mar 20, 2015 at 0:16
  • Autossh is better than mosh. I've used both. Sep 15, 2017 at 16:48
  • Here's a blog post by Jesse Keating, about using autossh with screen.
    – user61015
    Dec 29, 2017 at 23:37
20

tmux

This one is a classic. Use it whenever you run the risk of losing connection to a terminal.

$ tmux
$ sh do_something_that_takes_forever

!! Connection fails so you reconnect once you notice

$ tmux ls
0: 1 windows (created Tue Aug 23 12:39:52 2011) [103x30]

$ tmux attach -t 0

Just like that, you're back in action.

2
  • Thanks a ton! I was looking for how to do this with tmux, since that seems to be what most people consider as the better option.
    – CoolOppo
    Jul 28, 2016 at 21:09
  • 1
    much better then screen! Sep 13, 2020 at 20:22
20

Sometimes I forgot to run screen too and losing my unfinished work. In this case, though we can't reattach to a broken SSH session, reparenting a running program to a new terminal and resuming what you were doing is still possible thanks to reptyr.

After accidentally disconnected from a SSH session, the first thing first is to run screen (or tmux, whatever you like) lest the connection gets broken again. Then in the new session, run ps aux | grep {The process to be resumed} to get the PID. With the PID, you could try reptyr {PID} or reptyr -T {PID} (if there are subprocesses) to continue the work.

4
  • This worked great!
    – tr4nc3
    Jul 24, 2021 at 6:12
  • 4
    Very nice solution! Tip: If reptyr is not yet installed on the server, and the interrupted session that needs recovery is already a package management operation like apt upgrade, so that one cannot apt install reptyr just now: It is possible to scp just the reptyr executable file to the server and then run it like described here. Aug 29, 2021 at 18:00
  • I had a case where I was using tmux, the connection died but the login shell (fish) was in a broken state so the tmux sessions got lost when trying to reconnect. reptyr saved the day in a big way. May 9, 2023 at 10:50
  • This is exactly what I needed! And it was for an apt update process. So I used git clone https://github.com/nelhage/reptyr; cd reptyer; make install to install it. Thanks.
    – Twisty
    Jan 26 at 16:11
11

I'd install and start screen to fix your problem. Screen will let you reconnect to a previous screen session.

Apart from that, screen also let's you do cool things like split your screen, view the console etc. You can find more info here and here.

For starters, if you get disconnected, you can use

screen -ls

to view your sessions and

screen -r ${session} 

to reconnect to a disconnected one.

10

Use reptyr to reparent a running program to a new terminal.

HOW TO:

Lets assume that we have connected to ssh from some previous IP address aaa.bbb.ccc.old,
then run aptitude process, and in the middle of the work that connection got broken.

We login with ssh again (optionally from a different IP address aaa.bbb.ccc.new).
The current connection uses pty/1 pseudo-terminal:

$ tty
/dev/pts/1

But there are 2 open connections, the current pts/1 one, and the old broken one using pts/0:

$ who
user     pts/0        2022-02-22 22:00 (aaa.bbb.ccc.old)
user     pts/1        2022-02-22 22:11 (aaa.bbb.ccc.new)

Lets list all shells and their subprocesses:

$ ps auf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
user        2000  1.3  0.5  10168  5236 pts/1    Ss   22:11   0:00 -bash
user        2001  0.0  0.3  10620  3304 pts/1    R+   22:11   0:00  \_ ps auf
user        1000  0.0  0.5  10168  5132 pts/0    Ss   22:00   0:00 -bash
user        1001  5.8 16.0  46639 15839 pts/0    Sl+  22:00   0:03  \_ aptitude
root         820  0.0  0.1   5836  1840 tty1     Ss+  22:00   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root         810  0.0  0.2   7360  2260 ttyS0    Ss+  22:00   0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220

There are 2 bash shells:
the current pts/1 one with PID 2000 running ps auf
and the old broken pts/0 one with PID 1000 running aptitude

Lets reconnect to the old one with reptyr:

$ sudo reptyr -T 1000

This way we gain access to the old shell and its subprocesses.
Since aptitude uses ncurses, you may need to press Ctrl+L or resize the terminal emulator's window to force it to repaint itself.
Now we can quit aptitude, and then exit the old shell (1000),
thus returning to our new shell (2000) with only 1 connection remaining, which is the current one:

$ who
user     pts/1        2022-02-22 22:11 (aaa.bbb.ccc.new)
1
  • Very good tip about forcing the ncurses window to repaint itself.
    – Sam
    Sep 23, 2023 at 18:12
3

While screen will keep your shell session open on the remote server if your ssh session drops, it won't do anything about the problem of ssh connections being dropped. As zero_r suggests, try tuning your ssh connection with keep alives and long timeouts.

I suggest you track down the cause of the lost packets causing the problems and fix that instead of working around it.

2
  • Thanks for the suggestion. We are working on tracking down the root problem of packet loss, it's simply taking a while to figure it out (very tricky, this one!). This really is a work-around for this trouble period. One never knows when something will cause a dropped connection.
    – palehorse
    Jun 4, 2009 at 13:40
  • 1
    It will generate a lot of noise, but you can try running ssh through strace and see what it doing/reports when the ssh connection drops. It might just report what you already know, but who knows...
    – David
    Jun 4, 2009 at 15:42
3

a more modern alternative to screen, alas not available to some types of "virtualisation" (e.g. in cygwin you can have "screen", but not "tmux" due to the way it's designed), but wherever you have the option to install tmux, I'd strongly recommend to go for that one insted of screen.

1
3

This answer is still relevant, screen exists, is supported, still does the job, etc, but there are other alternatives now: tmux offers extra features, and both it and screen are more commonly (in my experience) used wrapped in Byobu. All three are available in the standard repositories of major Linux distributions.


As others have pointed out, screen is generally the best solution for this and it adds a host of other useful features too.

You can setup your profile on the remote machine to automatically start and/or reconnect to screen on login, which saves you forgetting to start screen the one time you need it because you suffer a connection drop.

See http://tlug.dnho.net/?q=node/239 (or search Google for many other examples dones slightly different ways).

0
2

Here's another non-screen solution.

Putty tray allows you to do just that, it has a reconnect on disconnect option that isn't in regular or putty-ng

https://puttytray.goeswhere.com/

it's a (completely open source) fork of putty with other options, go to connection settings and there are 2 options, one for 'attempt to reconnect on connection failure', and 'attempt to reconnect on startup'.

3
  • 3
    Ehh. Does it come with a keylogger and an automatic ppk exfiltration utility, by any chance? Jan 20, 2015 at 10:14
  • 2
    I'm assuming you're asking if it somehow, steals your details, it's completely open source, the source code is available on github if you'd like to look its github.com/FauxFaux/PuTTYTray there are quite a few contributors :)
    – munkiepus
    Jan 20, 2015 at 10:21
  • great cheers :)
    – munkiepus
    Jan 20, 2015 at 10:24
1

Yet another option is to run Jupyter Notebook, which has its own pretty good web-based terminal emulator.

Upsides

  • When I reattach my vpn I can re-open all my connections across several terminals on several hosts by opening all bookmarks in a folder.
  • I can open the terminal on any device and it looks decent on any device

Downsides

  • I do lose the console history (not the commands, just the scrolling) after reconnecting.

How to

  1. On some_host, run jupyter-notebook --no-browser --port=$SOME_PORT &.

  2. create session some_label in your browser by pointing to http://some_host:$SOMEPORT/terminals/some_label. You can now disconnect.

  3. Bookmark your sessions. When you reconnect, you can open them all at once. If the tabs are already open, you will need to refresh them.

If $SOME_PORT is not open on some_host, then use ssh to port-forward and connect to http://localhost:$SOME_LOCAL_PORT/terminals/some_label instead:

ssh -q -N -f -L localhost:$SOME_LOCAL_PORT:localhost:$SOME_PORT $some_host
1

My problem was simple to solve. On my /etc/sysconfig/network-scripts/ifcfg-eth0 was wrong with the parameter IPADDR set to a static IP and BOOTPROTO=dhcp. After updating the BOOTPROTO=static parameter the problem was solved.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .