1

I have a main backup server running rsnapshot, with ~2TB of backups stored on it. After the nightly backups, I copy the contents of the rsnapshot directory to an offsite server, using rsync -aH --delete /source /dest. However, this seems to copy the entire contents of the backup directory every night, as the offsite copy takes ~9 hours to complete.

I assume my rsync command doesn't have the right switches, but there may be something else I'm missing. Any ideas?

2
  • Rsnapshot uses special hierarchy of directories. Which one do you use as the source?
    – Ilya I
    Mar 25, 2014 at 6:55
  • I'm backing up the entire rsnapshot hierarchy (daily.0 through monthly.1). Should I just be backing up daily.0?
    – zymhan
    Mar 26, 2014 at 14:36

3 Answers 3

2

The issue here is that if you want to keep a copy of the whole versioned backup on a remote location, you're going to have to handle the fact that rsnapshot rotates the directory names whenever it runs, so you'll end up with a fresh copy of the most recent backup being copied over every time.

You can alter the way rsnapshot runs in terms of when it performs the copies and when it rotates (renames) the directories and you can also execute scripts at various points in rsnapshots execution cycle, like before or after a specific backup point is run.

There's a helpful post here:

https://stackoverflow.com/questions/10691522/mirror-rsnapshot-backup-directory

Another approach is to get a copy of rsnapshot on your second machine and have it backup the contents of /snapshot_root/daily.0 AFTER your main back has been completed. This, in conjunction with the correct rsync flags will give you an incremental copy of your latest backups.

1
  • That's a really good point, I hadn't considered that the names of the directories constantly change. I really like your suggestion, I'm gonna try that out!
    – zymhan
    Apr 9, 2014 at 19:50
4

I believe you just copying the same files multiple times. Hope these details on how does the rsnapshot work will help resolve the problem.

Here is the most important point:

When rsnapshot daily is run, it will rotate all the daily.X directories, then copy the contents of hourly.5 into daily.0.

hourly.0 will always contain the most recent snapshot, and daily.6 will always contain a snapshot from a week ago. Unless the files change between snapshots, the full backups are really just multiple hard links to the same files. Thus, if your /etc/passwd file doesn't change in a week, hourly.0/localhost/etc/passwd and daily.6/localhost/etc/passwd will literally be the same exact file.

This is how rsnapshot can be so efficient on space. If the file changes at any point, the next backup will unlink the hard link in hourly.0, and replace it with a brand new file.

5
  • I already knew that about rsnapshot, but somehow I didn't connect that to what's happening here. I guess that means I should be okay just rsync'ing the daily.0 folder?
    – zymhan
    Mar 26, 2014 at 15:58
  • Yes, according to this information you will get the most recent daily backup by copying daily.0 directory.
    – Ilya I
    Mar 26, 2014 at 16:04
  • Sounds good. I would prefer to mimic the revision history that exists on my main rsnapshot server, though. Is that possible?
    – zymhan
    Mar 26, 2014 at 17:10
  • Yep, just sync the entire hierarchy, what's the problem? rsync won't re-transfer files that haven't been touched since its last run anyway, or look at more advanced backup solutions that support remote backups initially.
    – Ilya I
    Mar 26, 2014 at 17:33
  • Well, that's what I thought, but that doesn't seem to be what's happening. The transfer occurs daily, takes hours to complete, and takes about the same amount of time each day.
    – zymhan
    Mar 27, 2014 at 2:06
1

There is an option called --link-dest which does what you need. When this parameter points to your latest backup it will only copy files from src to dest if src is different from the data already in your last backup (referenced by --link-dest) ; if the data is the same it will create a symlink instead thereby giving you a whole versioned backup.

I use this option to create daily snapshots locally on a separate disk, for remote backup you will need to modify it to ensure the symlinks will be created correctly on the remote system. I would recommend adding some rotation of old backups and an initial standard backup must be created to compare against.

today=$(date "+%Y-%m-%d-%H-%M-%S")
rsync -a --link-dest=/localbackups/latest/ /data/ /localbackups/${today}
cd /localbackups/
rm -rf /localbackups/latest
ln -s $today latest

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .