Rsync tasks: prioritize link-dest over file transfer

Question

I have a command which looks like this:

rsync -avp -e ssh /folder user@server:/backups/2 --link-dest=../1 Folder number increases every day.

This command runs every night, over my very very slow internet connection. If it did not complete during the night, the process will be killed because otherwise my connection is unusable during the day. This would work fine, if Rsync starts with creating hard-links, and then does the data-transfer.

Unfortunately, it looks like Rsync performs its tasks on alphabetical order. So sometimes it starts with transferring data and then afterwards it will create its hard links.

If the process does not complete overnight, it needs to upload even more data the next night, because the hardlinks were not created in folder 2!

Is there a way to make Rsync create the hard links first and then do the filetransfers?

Thanks a lot!

Answer 1

The answer is to add --delay-updates. This causes Rsync to build a list of all changes first. The target will then create all the hard links and create all folders. After that, the datatransfer will start.

rsync -avp --delay-updates -e ssh /folder user@server:/backups/2 --link-dest=../1

Answer 2

The --link-dest=../1 number should point to the most recent successful backup, not the most recent attempt.

Or, better yet, use multiple --link-dest flags, one for the most recent success, then additional ones for any failed attempts since the most recent success.

Quoting the man page:

Beginning in version 2.6.4, multiple --link-dest directories may
be provided, which will cause rsync to search the  list  in  the
order  specified  for  an exact match.

Combine this with --delay-updates (as Joel suggested) and you'll have a fairly robust system.

However as long as you have very limited bandwidth, your problems are only going to get worse over time as your data needs grow. I was in a situation like this once. The rsync just barely finished over night. A few weeks later it wasn't finishing with 24 hours. Since it was a daily since, you can see the problem! It might catch up over the weekend, but soon that wasn't enough. Eventually I just had to find a way to get more bandwidth.

If you are going to use this in a production system, I recommend that you log how many failures there are, and how many "failures since last success". Set up monitoring to collect this data and alert if the "failures since last success" grows to an unacceptable number (2? 5? It depends on your needs).