0

I am trying to move several TB from multiple hosts to other hosts. Each host will have 8 different rsync commands specifying different directory structures and I will be replicating this across nine different hosts.

Below is an example of what I'll be running on each host (different destinations but identical structures):

sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/engine_* --exclude 'db1' --exclude 'db2' 10.2.2.16:/tmp/prod/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/client --exclude 'db1' --exclude 'db2' 10.2.2.16:/tmp/prod/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/logs --exclude 'db1' --exclude 'db2' 10.2.2.16:/tmp/prod/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/backup* --exclude 'db1' --exclude 'db2' 10.2.2.16:/tmp/prod/RSYNC_TEMP

sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/engine_* --exclude 'db' 10.2.2.16:/tmp/QA/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/client --exclude 'db' 10.2.2.16:/tmp/QA/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/logs --exclude 'db' 10.2.2.16:/tmp/QA/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/QA/backup* --exclude 'db' 10.2.2.16:/tmp/QA/RSYNC_TEMP

We plan on running this manually every night from 5pm - 5am (12 hour timeout) until it's synced to the new hardware, with a final run done to account for any changes for when we cut over officially.

Obviously, I'd rather not manually start up 72 different rsync jobs every night, and I'd like to figure out a simple way of killing them early if necessary.

Does anyone have any ideas on what would be a good way to manage this task?

  • 1
    I found aggregating all rsync commands into one, setting a bandwidth limit below link speed & just letting it run continuously to be far easier than working around time-of-day restrictions and wasting bandwidth with many parallel transfers. – anx Jun 5 at 20:41
0

Are you looking for a way to script this? Will something like this work (issued from the destination with sudo -u svc_unix timeout 43200 preceding the script - also remove the echo syntax around the rsync command in line 17)?

#!/bin/bash

#Configs:
ip=("10.1.1.1" "10.1.1.2" "10.1.1.3" "10.1.1.4" "10.1.1.5" "10.1.1.6" "10.1.1.7" "10.1.1.8")
source_base="/tmp01/var/lib/data"
source_array=(engine client logs backup)
destination_base="/tmp"
destination="RSYNC_TEMP"
## Just match the elements of the array with the IP addresses in ${ip[]}
destination_array=(prod QA foo bar stuff stuff prod dev)
exclude_file="some.file"

echo "script start time: $(date)"

for i in $(seq 1 ${#ip[*]}); do
  echo "  start time for ${ip[$((i -1))]}: $(date)"
  echo "rsync -azqe ssh ${ip[$((i -1))]}:$source_base/${source_array[0]}_* :$source_base/${source_array[1]} :$source_base/${source_array[2]} :$source_base/${source_array[3]} --exclude-from=$exclude_file localhost:$destination_base/${destination_array[$((i -1))]}/$destination/"
  echo "done with files coming from ${ip[$((i -1))]}"
  echo "script end time: $(date)"
done
  • Scripting is likely the way I plan to go. Will this run them in parallel or will it wait for one to finish before continuing in the loop? – user2752794 Jun 5 at 20:42
  • serially - no parallel - you could also have a list of exclusions for each source in its own exclusion file, then make that argument another array. – JayRugMan Jun 5 at 20:51
  • you can open a screen session tab on the target for each source and run a script for each host in it's own tab. A .screenrc file in your home directory will make that easy to manage. – JayRugMan Jun 5 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.