I have a VPS running Ubuntu 16.04 with 2T storage where I'd like to put my data, which currently resides on a 1T flash drive. I'd like to be able to go back in history to retrieve older versions of files.

I've looked at "duplicity" but read a lot of stories about it being slow, and taking a lot of space because of the need to create frequent full backups.

Then, I arrived at "rdiff-backup" as a solution to this, but it doesn't handle resume very well. I wasn't able to even get the initial sync up to my server because of interruptions. I tried to first rdiff-backup to a local directory, and then rsync (-a) that to my VPS. But when I then tried to rdiff-backup from my original directory, it wouldn't work (weird error about too many incrementals).

I'm now considering rsnapshot, or maybe wrapping rdiff-backup in my own tool that can handle resumes better (by for example doing rdiff-backup incrementally directory per directory).

Should rdiff-backup work how I tried to make it work? (local dir 1 rdiff-> local dir 2, local dir 2 rsync-> remote dir, local dir 1 rdiff-> remote dir) And if not, would rsnapshot be a good alternative?


I strongly suggest you to use rsnapshot, configuring it to retry multiple times if a connection problem is detected. For example, to enable the multi-tries approach with 3 max tries, please add the following line to your rsnapshot.conf file:

rsync_numtries 3

The thing is that you want to backup 1 TB of data. This means that even after your first initial backup getting done, the incremental updates can be quite large, since all the tools you mentioned are working on the file system level, and transfer the whole new file if you changed it to the VPS.

Also some of those tools are definitely not being made for backing up large file bases. Rsnapshot is reliable in my opinion, and duplicty might also worth looking at if you want to encrypt or compress (or both) your stuff on the VPS, which of course means the backup needs more time.

Having said that, what you really need for a fast backup solution after the initial synchronization are incremental backups on block level.

This is something that comes with ZFS or Btrfs, the commands are ZFS send or btrfs send.

If Btrfs is your file system of choice (which I cannot recommend with valuable data yet), then for example this script here from Marc Merlin could be interesting, though there are similar around: http://marc.merlins.org/perso/btrfs/post_2014-03-22_Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive.html

If you do prefer ZFS instead, for example Znapzend might worth looking at: http://www.znapzend.org/

There are also companies around offering ZFS backup space for ZFS send/receive, like rsync.net (not affiliated with them nor customer and yes, despite the name they do offer ZFS send).

Please note that while it's fully possible to use tools like rsnapshot for this task, it's way more practical and faster to use ZFS/Btrfs send instead of it due to your big amount of data unless you have a very big internet line. Those two are technical superior to rsnapshot et.al.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.