Backup Ubuntu server with cron/rsync: Problems with resuming large files

Question

I currently have a dedicated Ubuntu backup server which wakes itself up from suspend (using rtcwake) daily. 5 minutes later a cron BASH job kicks in, using rsync to pull files from various network locations. If this BASH script completes, the last thing it does is put the server back to sleep until the following day (5 minutes before the backup job starts running again - you get the picture!).

There is an additional cron job which runs at 10:45pm to 'clean up' any running script/rsync processes, and then suspend the machine until the next day.

My problem/query is: the machine is on a slow link (802.11n wireless) and has to sync some pretty big media files. I am using rsync commands similar to this:

rsync -aPvz -e ssh --itemize-changes --modify-window=60 --exclude 'thumbs.db' /source/ /destination/

And in the case where rsync can't complete within the time the server is awake, I assumed (because of -P) that it would try to pick up where it left off.

What actually seems to happen is that I end up with multiple partial copies of the same file in the directory, but the copy never completes.

E.g:

.BigFile.mkv.EJtNSS (4.1GB)
.BigFile.mkv.KSUStW (3.7GB)
.BigFile.mkv.LSewSA (4.3GB)

Has anyone come across a similar scenario before? I'm very happy to change it all over if there's a piece of software that will do a similar job (including handling the sleep/wake). I'm new to BASH scripting, but up for trying any suggestions.

Cheers, Loz

Answer 1

Adding --inplace will help with big files greatly. It makes picking up after a failed or killed rsync faster because it doesn't use temporary files.

Answer 2

From your symptom, sync attempts were most likely unfinished due to slow link and the additional sync killing afterwards. In that case --inplace option is actually the worst option since it is not designed for network-bound situations. Quoting from the --inplace option:

WARNING: The file’s data will be in an inconsistent state during the transfer (and possibly afterward if the transfer gets interrupted) ...

You may want to check out another post on syncing very large file with rsync. (Hint: --append) Besides, it is odd that the --partial option you specified (-P includes --partial) doesn't pick up existing partial file. There is not enough info to determine the cause yet, but it can be checked via more verbose option (add more -v).