Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a few test scripts, each of which runs a test PHP app. Each script runs forever.

So, cat.sh, dog.sh, and foo.sh each run a PHP script. Each shell script runs the PHP app in a loop, so it runs forever, sleeping after each run.

I'm trying to figure out how to run the scripts in parallel at the same time. See the output of the PHP apps in the stdout/term window.

I thought, doing something like the following might work:

foo.sh > &2
dog.sh > &2
cat.sh > &2

In a shell script it would be sufficient, but it's not working.

foo.sh, runs foo.php once, and it runs correctly
dog.sh, runs dog.php in a never ending loop. it runs as expected
cat.sh, runs cat.php in a never ending loop
 *** this never runs!!!

It appears that the shell script never gets to run cat.sh. If I run cat.sh by itself in a separate window/term, it runs as expected.

Does anyone have any ideas on how I can get these scripts to run in parallel?

share|improve this question

3 Answers 3

Your current setup is running your commands sequentially. The first one runs, and completes, allowing the second one to start. The second one never ends, so the third one never gets a chance to start.

Try this:

foo.sh >foo.out 2>foo.err &
dog.sh >dog.out 2>dog.err &
cat.sh >cat.out 2>cat.err &

That will tell each command to run in the background, and send their output and error information to files. When a command is run in the background, it detaches from the current session, and allows the next command to run. It's important to put the ampersand (&) as the last character of the line. Otherwise, it might not get recognized correctly.

Alternately, have each of your shell scripts run the associated .php script you're apparently calling with the same semantics as above. In this scenario, your foo.sh might contain:

#!/bin/bash
php foo.php >foo.out 2>foo.err &

Finally, if you need true parallel operation, check out GNU Parallel.

Note: If you really want the output going to the screen/terminal, you can leave off the output redirection:

foo.sh &

Remember, though, that running multiple commands concurrently can cause confusing output (you may now know what output or errors came from which command), and if you logout, the output has nowhere to go.

share|improve this answer
    
Hi Chris. Thanks for the reply. given that my scenario has each shell test running forever, i really don't want to write the stdout/stderr to a file, rather, i would like to see the output/stderr just displyed to the screen... i thought running in the background, meant i didn't have to wait for completion before running the next script.. have i missed something.. || also as to the acceptance rate.. i've tried to delete a bunch of things in the past but never could... in other cases, the answers were totally off base and no help –  tom smith Mar 28 '12 at 0:24
    
Running in the background does mean you don't have to wait for completion before running the next script/command. That's what the & at the end of the command does. If you don't place the ampersand at the end, it doesn't mean the same thing. Redirecting output to a file (especially for background commands) is usually a good thing, unless they send logs somewhere else (like syslog). Otherwise, it's easy to get output mixed up between the multiple commands, since they're running at the same time. I'll update the answer to add a note about output to the terminal. –  Christopher Cashell Mar 28 '12 at 1:49

This can be done with parallel command.

Beautiful examples: parallel

[me@neo]<bash><~> 19
05:51 Tue Mar 05 > man -f parallel
parallel (1) - run programs in parallel

--

parallel -j 3 -- "sh foo.sh" "sh dog.sh" "sh cat.sh"

This will start 3 scripts in parallel, at the same time.

parallel sh -c "echo hi; sleep 2; echo bye" -- 1 2 3

This will start by default as many process of the same command, depending of cpu count.

parallel -j 3 sh -c "echo hi; sleep 2; echo bye" -- 1 2 3

This will force to execute 3 processes at the same time, even if you have just 2 cpu's.

Example:

[me@neo]<bash><~> 31 
06:09 Tue Mar 05 > parallel -j 10 sh -c "echo Hello World; sleep 3; echo Good morning" -- $(seq 1 10)

Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Hello World
Good morning
Good morning
Good morning
Good morning
Good morning
Good morning
Good morning
Good morning
Good morning
Good morning
share|improve this answer

Chris has the good general idea for most scripts: end each command with & and they'll go into the background. That said, here's my something I keep around for some higher-end tasks. Let me introduce you to my handy toolbox script, parallel-exec.pl

    #!/usr/bin/perl
    use threads;
    use Thread::Queue;

    my @workers;
    my $num_threads = shift;
    my $dbname = shift;
    my $queue = new Thread::Queue;

    for (0..$num_threads-1) {
            $workers[$_] = new threads(\&worker);
                    print "TEST!\n";
    }

    while ($_ = shift @ARGV) {
            $queue->enqueue($_);
    }

    sub worker() {
            while ($file = $queue->dequeue) {
                    system ('./parser.pl', $dbname, $file);
            }
    }

    for (0..$num_threads-1) { $queue->enqueue(undef); }
    for (0..$num_threads-1) { $workers[$_]->join; }

With some light tweaking, you can change the number of threads, provide an array of commands, etc. As this stands right now, it takes the number of threads specified on the command line, a database name, and a list of files to import, then spawns one instance for each file running the specified number at once.

share|improve this answer
2  
GNU Parallel provides very similar functionality, but that's a handy little bit of Perl. It never hurts to have multiple options in your toolbox. –  Christopher Cashell Mar 28 '12 at 15:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.