Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Task: run blocks consisting of 3-5 commands (in parallel/background). Example block:

dd if=/dev/urandom of=/mnt/1/x bs=1024 count=1024000000 &
dd if=/dev/urandom of=/mnt/2/x bs=1024 count=1024000000 &
dd if=/dev/urandom of=/mnt/3/x bs=1024 count=1024000000 &

When it's done, next block should run. I suppose, this can be done via lock files:

real_task1 real_param1 ; rm /var/lock/myscript/task1.lock

real_task2 real_param1 ; rm /var/lock/myscript/task2.lock


# loop
 # while directory isn't empty - wait... # build task files from some queue
 for i in 1 2 3; do touch /var/lock/myscript/task$i.lock ; done
 ./ &
 ./ &
 ./ &
 # if doesn't exits then exit, else loop waits for files to be deleted

A number of methods to check if the directory is empty can be found here, don't sure which to use;

Question: any better way to implement this?

P.S. Possible status reporting method:

 command && taskid ; taskid; rm /var/lock/myscript/taskN.lock
share|improve this question
Guys, just don't know what to say: all answers are correct, thanks! – kagali-san Apr 14 '11 at 22:03
up vote 4 down vote accepted

This is exactly what gnu parallel is designed for, so I strongly recommend you use it. In particular, look at running it as a semaphore:

for i in {1..4}
  echo running $i
  sem -j3 df dd if=/dev/urandom of=/mnt/$i/x bs=1024 count=1024000000 ";" echo done
# sem --wait waits until all jobs are done.
sem --wait
share|improve this answer

Perhaps some variation on this?

while true
  wait $pid1
  wait $pid2
  wait $pid3
share|improve this answer
But yep, GNU parallel is better if available. – Eduardo Ivanec Apr 14 '11 at 4:32

Do you have any particular reason not to use something like GNU parallel? If you must use bash, then consider methods like those described in this blog post (wait and named pipes are helpful here).

share|improve this answer

"wait" waits for all background jobs to complete. Sample:

sleep 30 & sleep 40 & sleep 120 & wait

It waits till all commands are completed, i.e. at least 120 seconds for this example.

Hope this helps.

share|improve this answer

We tried to use the GNU sem utility, as described by Phil Hollenback above, but found it too heavy-weight (300+ instances crippled the machine). I looked around for similar tools to implement a light-weight counting semaphore but couldn't find anything suitable.

So I implemented one myself using flock, it's called semaphoric.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.