Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a main backup server running rsnapshot, with ~2TB of backups stored on it. After the nightly backups, I copy the contents of the rsnapshot directory to an offsite server, using rsync -aH --delete /source /dest. However, this seems to copy the entire contents of the backup directory every night, as the offsite copy takes ~9 hours to complete.

I assume my rsync command doesn't have the right switches, but there may be something else I'm missing. Any ideas?

share|improve this question
    
Rsnapshot uses special hierarchy of directories. Which one do you use as the source? –  Ilya I Mar 25 at 6:55
    
I'm backing up the entire rsnapshot hierarchy (daily.0 through monthly.1). Should I just be backing up daily.0? –  WildVelociraptor Mar 26 at 14:36

3 Answers 3

up vote 2 down vote accepted

The issue here is that if you want to keep a copy of the whole versioned backup on a remote location, you're going to have to handle the fact that rsnapshot rotates the directory names whenever it runs, so you'll end up with a fresh copy of the most recent backup being copied over every time.

You can alter the way rsnapshot runs in terms of when it performs the copies and when it rotates (renames) the directories and you can also execute scripts at various points in rsnapshots execution cycle, like before or after a specific backup point is run.

There's a helpful post here:

http://stackoverflow.com/questions/10691522/mirror-rsnapshot-backup-directory

Another approach is to get a copy of rsnapshot on your second machine and have it backup the contents of /snapshot_root/daily.0 AFTER your main back has been completed. This, in conjunction with the correct rsync flags will give you an incremental copy of your latest backups.

share|improve this answer
    
That's a really good point, I hadn't considered that the names of the directories constantly change. I really like your suggestion, I'm gonna try that out! –  WildVelociraptor Apr 9 at 19:50

I believe you just copying the same files multiple times. Hope these details on how does the rsnapshot work will help resolve the problem.

Here is the most important point:

When rsnapshot daily is run, it will rotate all the daily.X directories, then copy the contents of hourly.5 into daily.0.

hourly.0 will always contain the most recent snapshot, and daily.6 will always contain a snapshot from a week ago. Unless the files change between snapshots, the full backups are really just multiple hard links to the same files. Thus, if your /etc/passwd file doesn't change in a week, hourly.0/localhost/etc/passwd and daily.6/localhost/etc/passwd will literally be the same exact file.

This is how rsnapshot can be so efficient on space. If the file changes at any point, the next backup will unlink the hard link in hourly.0, and replace it with a brand new file.

share|improve this answer
    
I already knew that about rsnapshot, but somehow I didn't connect that to what's happening here. I guess that means I should be okay just rsync'ing the daily.0 folder? –  WildVelociraptor Mar 26 at 15:58
    
Yes, according to this information you will get the most recent daily backup by copying daily.0 directory. –  Ilya I Mar 26 at 16:04
    
Sounds good. I would prefer to mimic the revision history that exists on my main rsnapshot server, though. Is that possible? –  WildVelociraptor Mar 26 at 17:10
    
Yep, just sync the entire hierarchy, what's the problem? rsync won't re-transfer files that haven't been touched since its last run anyway, or look at more advanced backup solutions that support remote backups initially. –  Ilya I Mar 26 at 17:33
    
Well, that's what I thought, but that doesn't seem to be what's happening. The transfer occurs daily, takes hours to complete, and takes about the same amount of time each day. –  WildVelociraptor Mar 27 at 2:06

There is an option called --link-dest which does what you need. When this parameter points to your latest backup it will only copy files from src to dest if src is different from the data already in your last backup (referenced by --link-dest) ; if the data is the same it will create a symlink instead thereby giving you a whole versioned backup.

I use this option to create daily snapshots locally on a separate disk, for remote backup you will need to modify it to ensure the symlinks will be created correctly on the remote system. I would recommend adding some rotation of old backups and an initial standard backup must be created to compare against.

today=$(date "+%Y-%m-%d-%H-%M-%S")
rsync -a --link-dest=/localbackups/latest/ /data/ /localbackups/${today}
cd /localbackups/
rm -rf /localbackups/latest
ln -s $today latest
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.