Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Is there a way I can do the following in a unix terminal:

  1. Start a long running process
  2. Add another long running process to start when the previous is done
  3. Repeat step 2 until I have queued the processes I need to have run

Reason I ask is that I have some long running stuff I need to do. I could put all the commands in a simple bash script and just execute that, but the problem is that I am not always sure exactly what I need to run. So if I start the script and then remember another one I should run, I need to hit ^C a bunch of times until all the processes are killed, edit the script and add my new process and then start the whole thing again.

What I specifically am doing right now is to copy a lot of large files on to various external hard drives, and since I don't know exactly which ones I need to copy and to where right now I'd like to start the ones I do know I need to copy, and then add to the queue as I figure out the rest.

Hope that made sense... Is anything like this possible?

share|improve this question

5 Answers 5

The shell is perfect for that.

  1. Start the first program.
  2. Press Ctrl-z to suspend the program.
  3. Start the next program.
  4. Repeat steps 2 and 3 to add more programs to the queue. All the programs will be in Suspended mode.
  5. Now run the queue like this:

    while fg; do :; done
    

    You may not suspend this while loop or exit bash until the queue is done.

If you will need to log out (e.g. the programs will run for many days) then consider running the above steps in screen.

share|improve this answer
    
This works well for two programs, but gets unwieldy fast after that. When you're repeating step 2 to add Program3, you have to press ctrl-z twice if Program1 is still running, and after that you end up with stopped jobs which don't run unless you call on them explicitly, or else which run in reverse order if you just keep typing fg. –  mattdm Dec 10 '10 at 4:54
    
With the updated version, the fg is doing nothing. That's what the "no job control" message is telling you. So that doesn't work so well either. –  mattdm Dec 10 '10 at 20:02
    
You are right again. Thanks! I re-wrote the instructions - now it's bullet proof. –  Aleksandr Levchuk Dec 10 '10 at 20:25

Not very elegant, but quick and dirty:

process1 &
while kill -0 $! ; do sleep 1; done && process2 &
while kill -0 $! ; do sleep 1; done && process3 &

You would need to substitute actual PIDs for $! if you've run intervening background jobs.

share|improve this answer

The answer to your general question is: Absolutely. In the early days of computing, batch processing was the only way to do anything, and even when multi-user interactive systems were invented, batch-processing capability was the norm for large jobs. And it's still commonly done today in medium and large-scale environments using systems like Sun Grid Engine or Torque.

However, that's probably overkill for what you need. You could set up a more lightweight system to run scripts in a serial queue, but I don't think that approach is particularly well-suited to your specific task. Presuming parallel copies to different drives are acceptable, I think I'd attack it like this:

  1. Create directory structure corresponding to your your target drives:

    ~/copysystem/drive1/ ~/copysystem/drive2/ ~/copysystem/drive3/

  2. Install Incron.

  3. Set up an incrontab entry for each of these directories, which runs your copy script automatically on IN_MOVED_TO.

  4. Make your script either a) kill any previous instances of the same script when it starts or b) use a mkdir-based lockfile and block until the lock is cleared.

Then, all you need to do is move files to the various ~/copysystem/drive# directories, and they're all copied magically to your destination.

Especially in case of 4a, you probably want to use rsync -aP to copy your files, so that you can restart partial transfers from the middle. (Possibly in combination with --remove-sent-files, if you want to get rid of the originals.)

If you want to skip the complication of using incron, you can still take advantage of making your scripts block on a lock file. That works something like this:

#!/bin/bash
LOCKDIR="/var/run/copysystem/copysystem.lock"

while ! mkdir $LOCKDIR ; do
  echo "waiting for lock"
  sleep 5
done
trap rmdir $LOCKDIR EXIT

rsync commands go here....

This works because mkdir is an atomic operation — if it succeeds, you know the directory didn't exist. That's important, because if you use something like ! -f && touch, there's a race condition. (Same with scanning the process table for rsync commands, or the like.)

share|improve this answer
    
Using that last script of yours: Can I create the script, fix the rsync command, save the script, run the script, edit the script, change the rsync command to something else, rinse and repeat? Like, once I have started a script, can I edit it, or will that mess up the running script? –  Svish Dec 10 '10 at 2:09
    
You'll get unpredictable results that way. Somewhat surprisingly, shell scripts are read line by line as they're executed. So, don't do that. But that seems like a bad approach anyway. If there's something that's changed (a filename or destination location) pass it in as an argument. –  mattdm Dec 10 '10 at 3:20
    
oh, so does that mean I could actually edit the script while it is running and change where it goes? Like add things or remove things at the end of the script, while it is working on something earlier in the script? –  Svish Dec 10 '10 at 21:16
    
Well, like I said, you'll get unpredictable results. It's really not meant to work that way. –  mattdm Dec 10 '10 at 21:20

If this is something you'll be doing regularly, then its worth setting up some sort of scheduler to manage it for you.

I like the elegance of Aleksandr's solution - and it addresses all the points you originally raised - but I can see it does have limitations.

I've previously used the BSD lpd as a method for queueing jobs - the 'printer driver' is just a shell script so its easy to adapt to different tasks (in my case that meant managing 4 modems for polling data, sending faxes, SMS and other stuff).

share|improve this answer

What you want is a non-interactive command queue. Good news! I wrote one for you.

enqueue_cmd:

#!/usr/bin/perl -w

# Enqueue a job

use strict;

use Fcntl qw(:flock);

my $JOB_QUEUE_FILE = '/var/tmp/job_queue';
my $LOCK_TRIES = 5;
my $LOCK_SLEEP = 1;

my $jq;
open $jq, ">> $JOB_QUEUE_FILE" or die "!open $JOB_QUEUE_FILE: $!";

my $locked = 0;

LOCK_ATTEMPT: for (my $lock_tries = 0; $lock_tries < $LOCK_TRIES;
    $lock_tries++)
{
    if (flock $jq, LOCK_EX) {
            $locked = 1;
            last LOCK_ATTEMPT;
    }

    sleep $LOCK_SLEEP;
}

$locked or die "could not lock $JOB_QUEUE_FILE";

for (@ARGV) {
    print $jq "$_\n";
}

close $jq;

dequeue_cmd:

#!/usr/bin/perl -w

# Dequeue a jobs and run them

use strict;

use Fcntl qw(:seek :flock);
use FileHandle;

my $QUEUE_FILE = '/var/tmp/job_queue';
my $OUTPUT_FILE = '/var/tmp/job_out';
my $LOCK_TRIES = 5;
my $LOCK_SLEEP = 1;
my $JOB_SLEEP = 1;

my $locked;

my $jo;
open $jo, ">> $OUTPUT_FILE" or die "!open $OUTPUT_FILE: $!";
$jo->autoflush(1);

my $jq;
open $jq, "+< $QUEUE_FILE" or die "!open $QUEUE_FILE: $!";

my @jobs = ( );
my $job;

JOB: while (1) {

    if (-s $QUEUE_FILE == 0) {
            sleep $JOB_SLEEP;
            next JOB;
    }

    $locked = 0    

    LOCK_ATTEMPT: for (my $lock_tries = 0; $lock_tries < $LOCK_TRIES;
            $lock_tries++)
    {
            if (flock $jq, LOCK_EX) {
                    $locked = 1;
                    last LOCK_ATTEMPT;
            }

            sleep $LOCK_SLEEP;
    }
    $locked or die "could not lock $QUEUE_FILE";

    seek $jq, 0, SEEK_SET or die "could not seek to start of file";

    push @jobs, <$jq>;

    truncate $jq, 0 or die "could not truncate $QUEUE_FILE";
    seek $jq, 0, SEEK_SET or die "could not seek to start of $QUEUE_FILE";

    flock $jq, LOCK_UN;

    for $job (@jobs) {
            chomp $job;
            print $jo "## executing $job\n";
            print $jo `$job`;
    }

    sleep $JOB_SLEEP;
}

First, run nohup ./dequeue_cmd &, Then add your commands like so:

./enqueue_cmd "echo 'hello world'" "sleep 5" "date"
./enqueue_cmd "ls /var/tmp"

The output appears in /var/tmp/job_out:

tail -F /var/tmp/job_out
## executing echo 'hello world'
hello world
## executing sleep 5
## executing date
Fri Dec 10 16:35:43 PST 2010
## executing ls /var/tmp
ff
job_out
job_queue
ss
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.