I'm trying to syncronize files from a remote server that is not reliable, meaning the connection tends to fail "randomly" with
rsync: connection unexpectedly closed
Rsync is called with --partial, so I'd like to be able to call rsync in a loop until files are fully transfered. There doesn't seem to be a flag to tell rsync to retry.
What would be the best way to script it? A bash for loop?
If you are syncing everything in one sync, call rsync in a loop until rsync gives you a successful return code.
Something like:
RC=1
while [[ $RC -ne 0 ]]
do
rsync -a .....
RC=$?
done
This will loop, calling rsync, until it gives a return code of 0. You may want to add a sleep in there to keep from DOSing your server.
RC=12; while [[ $RC -eq 12 || $RC -eq 30 ]] (or whatever exit statuses you're seeing from network drops).
Jan 2, 2010 at 23:09
I ran into this same problem a while back. In the end I wrote something similar to David's answer, but gussied it up a little with max retries, responding to Ctrl-C, and such: http://blog.iangreenleaf.com/2009/03/rsync-and-retrying-until-we-get-it.html.
The obvious solution is to check the return value, and if rsync returns anything but success, run it again. Here was my first try:
while [ $? -ne 0 ]; do rsync -avz --progress --partial /rsync/source/folder [email protected]:/rsync/destination/folder; done
The problem with this is that if you want to halt the program, Ctrl-C only stops the current rsync process, and the loop helpfully starts another one immediately. Even worse, my connection kept breaking so hard that rsync would quit with the same "unkown" error code on connection problems as it did on a SIGINT, so I couldn't have my loop differentiate and break when needed. Here is my final script:
#!/bin/bash
### ABOUT
### Runs rsync, retrying on errors up to a maximum number of tries.
### Simply edit the rsync line in the script to whatever parameters you need.
# Trap interrupts and exit instead of continuing the loop
trap "echo Exited!; exit;" SIGINT SIGTERM
MAX_RETRIES=50
i=0
# Set the initial return value to failure
false
while [ $? -ne 0 -a $i -lt $MAX_RETRIES ]
do
i=$(($i+1))
rsync -avz --progress --partial /rsync/source/folder [email protected]:/rsync/destination/folder
done
if [ $i -eq $MAX_RETRIES ]
then
echo "Hit maximum number of retries, giving up."
fi
purtting it all together with sshpass
while ! sshpass -p 'xxxx' \
rsync --partial --append-verify --progress \
-a -e 'ssh -p 22' /source/ [email protected]:/dest/; \
do sleep 5;done
Just another take - a simple one liner to run indefinitely until succeeded, with a 1 second delay after a failure.
while true; do rsync -avz --info=progress2 --partial src/ dsr/ && break || sleep 1; done
And below a slight modification limiting to 10 failures.
for i in {0..10}; do rsync -avz --info=progress2 --partial src/ dsr/ && break || sleep 1; done
