Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 72 down vote favorite
38

I'm connecting to a linux machine through SSH, because I'm trying to run a heavy bash script that makes filesystem operations, and it's expected to keep running for hours. But I cannot leave the SSH session open because of internet connections issues I have. I doubt that running the script with the background operator, the ampersand &, will do the trick, because I tried it and later found that process was not completed. How can I logout and keep the process running?

share|improve this question

9 Answers

up vote 71 down vote accepted

The best method is to use screen.

Screen is a virtual terminal which you can run from a "real" terminal (actually all terminals today are virtual). Screen will keep running even if your ssh session gets disconnected. Any programm which you start from screen will keep running in that screen session. When you reconnect the ssh session, you can reconnect to the screen session and everything will be as if nothing happened, other than the time which passed.

The disadvantage of screen is the steep learning curve. Here is a nice tutorial with diagrams explaining the logical structure.

An alternative to screen is tmux.

Another method is to use nohup.

Programs started with nohup will ignore the HUP signal which is sent when the controlling terminal is closed. This happens when you disconnect your ssh session (or get disconnected). Although the program will keep running this way, you can no longer interact with it because it is no longer attached to any terminal.

nohup is usefull mainly for long running batch processes which, once started, no longer need any attention.

As an alternative to nohup, the bash builtin disown can be used to tell bash to not send the program the HUP signal.

share|improve this answer
3  
I like this answer as it provides a solution for both interactive and non-interactive situations. In the interactive case, screen gives you a lot more options, but if you are using authorized_keys to allow people to run a script remotely via ssh, the nohup option is a nice simple way for the script to start processes which last longer than the ssh session used to start them. – Mark Booth Sep 15 '11 at 10:33
lesmana, and @Mark Booth I like your answers as it tells more useful tips.. 1+ – rahmanisback Sep 15 '11 at 17:17
1  
@rahmanisback - Remember that you can change your accepted answer at any time. Just because so far EricA's answer is voted most highly doesn't mean it's the best answer for you, indeed making it your accepted answer may well encourage more people to vote it up as a good answer. – Mark Booth Sep 15 '11 at 23:02
1  
*cough*tmuxisbetter*cough* – crasic Sep 16 '11 at 0:07
1  
@TheLQ - byobu is GNU Screen. You're still using screen, just with a highly-customized .screenrc. – EEAA Sep 16 '11 at 21:34
show 5 more comments
up vote 75 down vote

There are a few ways to do this, but the one I find most useful is to use GNU Screen.

After you ssh in, run screen. This will start another shell running within screen. Run your command, then do a Ctrl-a d.

This will "disconnect" you from the screen session. At this point, you can log out or do anything else you'd like.

When you want to re-connect to the screen session, just run screen -RD from the shell prompt (as the same use user that created the session).

share|improve this answer
Awesome! Neat answer, thanks... – rahmanisback Sep 15 '11 at 6:20
4  
Screen has a whole bunch of command, all starting with Ctrl-a. If you only learn one extra, start with "Ctrl-a ?". Then you'll wan't to learn "Ctrl-a c" and "Ctrl-a n" – olafure Sep 15 '11 at 12:06
@olafure +1, thanks. Looks like Screen is going to be of my primary toolbox. – rahmanisback Sep 15 '11 at 17:11
tmux > screen. Try it, you'll never go back. – h0tw1r3 Sep 16 '11 at 15:31
+1 for tmux. I gave up screen 5 weeks ago. – Bryan Hunt Jun 6 '12 at 8:46
up vote 32 down vote

In bash, the disown keyword is perfectly suited to this. First, run your process in the background (either use &, or ^Z then type bg):

$ wget --quiet http://server/some_big_file.zip &
[1] 1156

By typing jobs you can see that the process is still owned by the shell:

$ jobs
[1]+  Running  wget

If you were to log out at this point, the background task would also get killed. However, if you run disown, bash detaches the job and allows it to continue running:

$ disown

You can confirm this:

$ jobs
$ logout

You can even combine the & and the disown on the one line, like so:

$ wget --quiet http://server/some_big_file.zip & disown
$ logout

Better than running nohup in my opinion because it doesn't leave nohup.out files littered all over your filesystem. Also, nohup must be run before you run the command — disown can be used if you only decide later on that you want to background and detach the task.

share|improve this answer
1  
That's a very good answer, 1+. The only preference for nohup or Screen would be the independence of bash, and might be used with other shell. But I will stick to your approach whenever I'm using bash. – rahmanisback Sep 15 '11 at 17:34
1  
Nice looking avatar :) – rahmanisback Sep 15 '11 at 17:55
Yes – this is bash–specific, as bash is the only shell I've ever used. I wonder if other shells support anything similar (i.e. launching in background without nohup) – it would be fantastic if someone could post other answers for other shells. – Jeremy Visser Sep 16 '11 at 2:46
1  
see my answer showing how to do it it with any sh-lookalike – w00t Sep 16 '11 at 16:49
up vote 31 down vote

The tool nohup, available on most Linux boxes will do this.

share|improve this answer
1  
This is by far the simplest answer. Any output from is automatically directed to nohup.out and can be examined later. – Julian Sep 15 '11 at 10:01
3  
nohup does not require zsh at all. – Aaron Brown Sep 15 '11 at 11:55
nohup is the correct answer, it is short for no hangup. – BinaryNights Sep 15 '11 at 19:18
2  
nohup is found on far more machines then screen, so you should know how to use it. – Zenon Sep 15 '11 at 21:11
up vote 21 down vote

Just to be thorough, I'll point out tmux, which has the same basic idea as screen:

tmux is intended to be a modern, BSD-licensed alternative to programs such as GNU screen. Major features include:

  • A powerful, consistent, well-documented and easily scriptable command interface.
  • A window may be split horizontally and vertically into panes.
  • Panes can be freely moved and resized, or arranged into preset layouts.
  • Support for UTF-8 and 256-colour terminals.
  • Copy and paste with multiple buffers.
  • Interactive menus to select windows, sessions or clients.
  • Change the current window by searching for text in the target.
  • Terminal locking, manually or after a timeout.
  • A clean, easily extended, BSD-licensed codebase, under active development.

It is, however, approximately infinitely easier to search for on Google.

share|improve this answer
2  
Using "gnu screen" as your search query works quite well. – gnur Sep 15 '11 at 7:45
2  
+1000 for tmux! – mbq Sep 15 '11 at 12:29
up vote 9 down vote

Screen is overkill for just keeping processes running when you logout.

Try dtach:

dtach is a program written in C that emulates the detach feature of screen, which allows a program to be executed in an environment that is protected from the controlling terminal. For instance, the program under the control of dtach would not be affected by the terminal being disconnected for some reason.

dtach was written because screen did not adequately meet my needs; I did not need screen's extra features, such as support for multiple terminals or terminal emulation support. screen was also too big, bulky, and had source code that was difficult to understand.

screen also interfered with my use of full-screen applications such as emacs and ircII, due to its excessive interpretation of the stream between the program and the attached terminals. dtach does not have a terminal emulation layer, and passes the raw output stream of the program to the attached terminals. The only input processing that dtach does perform is scanning for the detach character (which signals dtach to detach from the program) and processing the suspend key (which tells dtach to temporarily suspend itself without affecting the running program), and both of these can both be disabled if desired.

Contrary to screen, dtach has minimal features, and is extremely tiny. This allows dtach to be more easily audited for bugs and security holes, and makes it accessible in environments where space is limited, such as on rescue disks.

share|improve this answer
Thank you. I came here just to post about dtach too. It's my go-to thing for terminal detachment now; screen does WAY too much, and interferes with input to a ridiculous degree. The fact screen needs its own termcap is pretty unsettling. – fluffy Sep 15 '11 at 18:12
Thanks. I will install it and give it a try. – rahmanisback Sep 15 '11 at 22:32
up vote 7 down vote

"byobu" on Ubuntu is a nice front-end to screen, by pressing "Ctrl-?" you get a list of all the keyboard shortcuts. It adds a status bar that can be useful for watching CPU load, disk space, etc. Overall it provides an experience that I would describe as a terminal based VNC connection.

"nohup" allows starting a job in the background with it's output routed to a log file, which can always be redirected to /dev/null if not required.

share|improve this answer
up vote 6 down vote

The at command can be useful for this kind of situation. For example, type:

at now

And you can then enter a command or series of commands that will be run. The results should be e-mailed to you, if e-mail is set up correctly on the machine.

Instead of now, you can specify a time optionally with a date, or an time expression like now + 15 minutes. See man at for more details.

share|improve this answer
up vote 5 down vote

Here's a way to daemonize any shell process, no external programs needed:

( while sleep 5; do date; done ) <&- >output.txt &

When you then close your session, the job will continue to run as evidenced by the output.txt file (which has buffering so it takes a while to show non-zero). Don't forget to kill your job after testing :-)

So all you need to do is close stdin and background the job. To be really good, first cd to / so you don't hold on to a mount.

This works even in simple sh under Solaris.

share|improve this answer
Interesting. That emerged some notable questions. I can see that you set the STDIN to nothing, the - operator? What is the difference between < /dev/null and &- ? I guess that STDIN (and others STDOUT and STDERR) could be assigned either a file by < file, or a stream by <& stream in case of STDIN. Would it be the same using < /dev/null in your example above? And Does the operator - above refer to a null as the stream? – rahmanisback Sep 16 '11 at 22:29
When you do x<&-, that closes file descriptor x. In this case there is no x, which makes bash default to 1, i.e. standard input. If you use </dev/null you're not closing stdin, you're just giving an empty file to the program as input. – w00t Sep 19 '11 at 17:15
1  
And to be honest I don't really know why this daemonizes the thing you're running :-) It does work though, we use it in production. I discovered it while messing around hoping I could daemonize a process in shell without needing anything special - so I started by closing stdin and that was enough. I should go read some shell sources but I presume that if you close stdin and background the process, it also detaches the process. – w00t Sep 19 '11 at 17:19
1  
I think the reason is that a SIGHUP (the actual signal that causes the child to quit when the shell dies) is triggered when a parent process closes its child's stdin handle. However, if stdin starts out as null, rather than being closed after the fact, there is no way for the parent to trigger the SIGHUP. Nice find, though — never would have thought of that. – Jeremy Visser Oct 15 '11 at 14:06
@JeremyVisser that sounds really plausible! – w00t Oct 17 '11 at 8:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.