3

I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC.

The below shell script will copy the files into TEST1 and TEST2 directory in machineA..

#!/bin/bash
set -e

readonly TEST1=/data01/test1
readonly TEST2=/data02/test2
readonly SERVER_LOCATION=(machineB machineC)
readonly FILE_LOCATION=/data/snapshot

dir1=$(ssh -o "StrictHostKeyChecking no" david@${SERVER_LOCATION[0]} ls -dt1 "$FILE_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${SERVER_LOCATION[1]} ls -dt1 "$FILE_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    rm -rf $TEST1/*
    rm -rf $TEST2/*
    for el in $test1_partition
    do
        scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$el"_200003_5.data $TEST1/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$el"_200003_5.data $TEST1/.
    done
    for sl in $test2_partition
    do    
        scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$sl"_200003_5.data $TEST2/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$sl"_200003_5.data $TEST2/.
    done
fi

https://unix.stackexchange.com/questions/53390/is-there-a-way-to-run-process-parallelly-in-the-loop-of-a-bash-script

Currently it copies the file from machineB and machineC into machineA TEST1 directory first, and if it is done, then only it will go and copy the files from machineB and machineC into machineA TEST2 directory.. Is there any way I transfer the files both in TEST1 and TEST2 directory simultaneously?

I am running Ubuntu 12.04

3

Replace

done

with

done &

This will start the two for-loops in the background.

And finish the script with:

wait

This will wait for the two for-loops to complete.

  • Thanks for suggestion. What does wait do here? It will wait for the script to finish before exiting out from the script correct? – arsenal Feb 15 '14 at 15:11
  • And the file transfer will happened in the background. It's just that, control won't come out of the script? – arsenal Feb 15 '14 at 15:30
  • 1
    wait will wait for all child processes to exit. – Michael Martinez Mar 13 '14 at 17:37
0

Try something like this.

PROCESSES=4  # Number of threads you want
for item in ${LIST}; do
    ACTION ${item} &
    while (( $(jobs |wc -l) >= (( ${PROCESSES} + 1 )) )); do
        sleep 0.1
    done
done
wait

This will work through your items in LIST and execute ACTION against item in the background.

Then it checks to see how many jobs are still running and waits until you're not threadcapped to continue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.