improving rsync backup performance

Question

What are the best techniques to improve rsync over ssh mirroring between unix boxes, assuming that one system will always have the master copy and the other system will always have a recent copy (less than 48hrs old)

Also, what would one have to do to scale that approach to handle dozens of machines getting a push of those changes?

Answer 1

If :

• The modification time of your files are right
• The files are not really big
• No push can be missed (or there is some kind of backlog processing)

You can use find -ctime or file -cnewer to make a list of changed file since the last execution, and copying over only the modified files (Just a glorified differential push).

This translated itself quite nicely for multiple hosts : just do a differential tar on the source, and untar it on all the hosts.

It gives you something like that :

find -type f -cnewer /tmp/files_to_send.tar.gz > /tmp/files_to_send.txt
tar zcf /tmp/files_to_send.tar.gz --files-from /tmp/files_to_send.txt 
for HOST in host1 host2 host3 ...
do
    cat /tmp/files_to_send.tar.gz | ssh $HOST "tar xpf -"
done

The script has te be refined, but you get the idea.

Answer 2

Presuming that the data you're rsyncing isn't already compressed, turning on compression (-z) will likely help transfer speed, at the cost of some CPU on either end.

Answer 3

Another strategy is to make ssh and rsync faster. If you are going over a trusted network(read: private), then encrypting the actual payload is not necessary. You can use HPN ssh. This version of ssh only encrypts authentication. Also, rsync version 3 starts transfering files while building the file list. This of course is a huge time savings over rsync version 2. I don't know if that's what you were looking for, but I hope it helps. Also, rsync does support multicasting in some way, though I will not pretend to understand how.

Answer 4

If you're transferring very large files with lots of changes, use the --inplace and --whole-file options, I use these for my 2Gb VM images and it helped a lot (mainly as the rsync protocol wasn't doing much with passing incremental data with these files). i don;t recommend these options for most cases though.

use --stats to see how well your files are being transferred using the rsync incremental protocol.

Answer 5

When you are rsyncing as a backup method, the biggest problem you will run into is going to be if you have a lot of files you are backing up. Rsync can handle large files without a problem but if the number of files you are backing up gets too large then you will notice that the rsync won't complete in a reasonable amount of time. If this happens you will need to break the backup down into smaller parts and then looping over those parts e.g.

find /home -mindepth 1 -maxdepth 1 -print0 | xargs -0 -n 1 -I {} -- rsync -a -e ssh {} backup@mybackupserver:/backup/

or tarring the fileset down to reduce the number of files.

As for having dozens of machines getting a mirror of those changes, it depends on how fresh the backup needs to be. One approach would be to mirror the changes from the primary server to the backup server and then have the other servers pull their changes off the backup server either by an rsync daemon on the initial backup server and then either scheduling the other servers to pull at slightly different times or by having a script use passwordless ssh to connect to each of the servers and tell them to pull a fresh copy of the backup which would help prevent overwhelming your initial backup server - but whether you go to that much trouble is going to depend on how many other machines you have pulling a copy of the backup.