How to parallelize the for loop while scp the files?

Question

I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC.

The below shell script will copy the files into TEST1 and TEST2 directory in machineA..

#!/bin/bash
set -e

readonly TEST1=/data01/test1
readonly TEST2=/data02/test2
readonly SERVER_LOCATION=(machineB machineC)
readonly FILE_LOCATION=/data/snapshot

dir1=$(ssh -o "StrictHostKeyChecking no" david@${SERVER_LOCATION[0]} ls -dt1 "$FILE_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${SERVER_LOCATION[1]} ls -dt1 "$FILE_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    rm -rf $TEST1/*
    rm -rf $TEST2/*
    for el in $test1_partition
    do
        scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$el"_200003_5.data $TEST1/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$el"_200003_5.data $TEST1/.
    done
    for sl in $test2_partition
    do    
        scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$sl"_200003_5.data $TEST2/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$sl"_200003_5.data $TEST2/.
    done
fi

https://unix.stackexchange.com/questions/53390/is-there-a-way-to-run-process-parallelly-in-the-loop-of-a-bash-script

Currently it copies the file from machineB and machineC into machineA TEST1 directory first, and if it is done, then only it will go and copy the files from machineB and machineC into machineA TEST2 directory.. Is there any way I transfer the files both in TEST1 and TEST2 directory simultaneously?

I am running Ubuntu 12.04

Answer 1

Replace

done

with

done &

This will start the two for-loops in the background.

And finish the script with:

wait

This will wait for the two for-loops to complete.

Answer 2

Try something like this.

PROCESSES=4  # Number of threads you want
for item in ${LIST}; do
    ACTION ${item} &
    while (( $(jobs |wc -l) >= (( ${PROCESSES} + 1 )) )); do
        sleep 0.1
    done
done
wait

This will work through your items in LIST and execute ACTION against item in the background.

Then it checks to see how many jobs are still running and waits until you're not threadcapped to continue.