I need to scp the files from machineB and machineC to machineA. I am running my below shell script from machineA. I have setup the ssh keys properly.

If the files are not there in machineB, then it should be there in machineC.


readonly PRIMARY=/data01/primary/.
readonly FILERS_LOCATION=(machineB machineC)
readonly MAPPED_LOCATION=/bat/data/snapshot
PARTITION=(0 3 5 7 9)


scp david@${FILERS_LOCATION[0]}:$dir1/weekly_1980_[$el]_200003_5.data $PRIMARY || scp david@${FILERS_LOCATION[1]}:$dir2/weekly_1980_[$el]_200003_5.data $PRIMARY

Now my question is if you take a look into my above scp command, I have [$el] for now (which is wrong), I need to replace this with PARTITION, which means I need to iterate PARTITION and replace [$el] with each number in PARTITION.

So my scp command should look like this if I iterate PARTITION one by one -

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_0_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_0_200003_5.data /data01/primary/.

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_3_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_3_200003_5.data /data01/primary/.

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_5_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_5_200003_5.data /data01/primary/.

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_7_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_7_200003_5.data /data01/primary/.

scp david@machineB:/bat/data/snapshot/20131222/weekly_1980_9_200003_5.data /data01/primary/. || scp david@machineC:/bat/data/snapshot/20131222/weekly_1980_9_200003_5.data /data01/primary/.

Problem Statement:-

  1. How do I iterate PARTITION in such a way, so that I can make the above SCP command?
  2. And also, as you can see, I am copying the files one by one into machineA /data01/primary/ folder. Is there any better way of doing this? Meaning, can I copy all the files together in one shot instead of doing it one by one?

You can iterate over the PARTITION array like this

for el in "${PARTITION[@]}"
    echo "$el"
  • @lain: Thanks a lot. Yes it works fine. But I also have my second question... With this way, I will be doing file transfer one by one.. Is there any way, I can make this more efficient so that I can move all the files at once? Or some better way?
    – arsenal
    Dec 24 '13 at 10:39
  • @TechGeeky: In what way do you want to improve efficiency? What do you have spare capacity to do ?
    – user9517
    Dec 24 '13 at 13:01
  • By efficient I meant to say, Is moving the files one by one is the only way? Can you move chunk of files in one shot? I am just trying to think whether this is the only way to move the files? Because, I need to movie atleast 420 files at a time, so it will take some time..
    – arsenal
    Dec 24 '13 at 19:29
  • @TechGeeky: Surely you need to determine if your scp command is saturating your network/IO before you can decide if there is capacity to run the commands in parallel ?
    – user9517
    Dec 24 '13 at 19:33
  • I guess, it should be fine with less load for now.. I can test it and see whether that will cause any problem or not? In any case, if I need to make this parallel then how I am going to do it? Any thoughts?
    – arsenal
    Dec 24 '13 at 19:41

I would advise looking at rsync to complete this task, as it provides greater functionality all-round for this type of backup. You could execute this from machineA, pointing it at the snapshot directories on B & C, and your whole snapshot directory would be mirrored.

You could then use cron to schedule this at set intervals. scp is ideal for one-off copies, but your scenario sounds like a regular backup job. rsync also communicates over standard SSH ports, so if scp works at present there should be no need for changes to the network, firewalls, etc.

With rsync you could even implement functionality to create an identical mirror on all three servers if that was required.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.