I have a large amount of files in both quantity and total file size. (we're talking a few terabytes). I would like to sync these files/folders once to an external backup system, then run a daily task that will resync the backup based on the daily changes. The changes aren't that frequent but some days we might have around a 300GB differential (for about 1.5K files).
I've been considering rsync or rdiff-backup or rsnapshot as some tools but I've wanted to run some tests with rsynch first. I've had one major issue with rsync and that is:
Checking of existing files for changes takes way too long. We're talking over 20h which makes the daily backup pointless. This is using rsync -rvhzP or -rvhP. It seems to simply scan all files and takes hours on end even if no file was added/changed/deleted.
Am I doing something wrong? Will any of the other systems I mentioned (rdiff-backup or rsnapshot) perform any better? I was going under the assumption they were based off of rsync anyways.
Thanks in advance.
Update with extra information: We have about 2600 directories and 100k files totalling around 3.5TB, ran the tests using
rsync version 3.0.9 protocol version 30. As far as daily changes go, there are generally 10- file changes a day but it can peak at arround 1.5K file changes/additions/deletions and about 300Gb in volume (though these peaks are'nt that frequent, and generaly spread appart)