Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

How do I backup 50GB+ of files on a linux box to a remote sftp account.

My requirements

  • Store several versions of the backups.
  • Encrypted before sent to remote server.
  • Disk and bandwidth efficent, think incremental.
  • Using simple linux commands (preferred support by Busybox on a Synology Diskstation)

My solution

Currently I'm using rsnapshot to make one local backup every day. Then storing weekly, monthly backups in the rsnapshot way. Then doing a full backup every month with tar, which I encrypt with gpg2 before sending to my friend.

Then I do an incremental level 1 backup using the snapshot file from the full backup.

But unfortunately this doesn't really work. Because each time rsnapshot is executed, my files get new dates and maybe other stuff. And it looks like tar only uses date to verify if the file has changed or not.

Questions

  • Can I solve my backup problem with this solution?
    • One way to do it, is to backup the original files instead of the rsnapshot/rsync copy. But currently I don't really trust tar, maybe it will miss something else. I would like something that does a checksum or some other more performance friendly magic.
  • Should I use some other software?

Code example

This is a simplified version of my script, that illustrates the problem.

cd ~
mkdir backup-test
cd backup-test

mkdir data
mkdir rsync-copy
mkdir backup

# Create 10 10mb files
for i in 0 1 2 3 4 5 6 7 8 9 
do
    dd if=/dev/urandom of=data/file$i.txt bs=1048576 count=10
done

# Simulate rsnapshot of my files.
rsync data/* rsync-copy/

# Make full backup
/opt/bin/tar c -g backup/full.snar rsync-copy --file backup/full.tar

# Simulate rsnapshot of my files again. No changes in my files.
rsync data/* rsync-copy/

# Make incremental backup, this will create a new full copy.
cp backup/full.snar backup/inc.snar
/opt/bin/tar c -g backup/inc.snar rsync-copy --file backup/inc.tar

# Make incremental backup, this will not create a new full copy.
# Because we haven't executed rsync, which sets dates and stuff.
cp backup/inc.snar backup/inc2.snar
/opt/bin/tar c -g backup/inc2.snar rsync-copy --file backup/inc2.tar
share|improve this question

closed as off-topic by Tim Brigham, Ward, Falcon Momot, Stephane, TheCleaner Sep 20 '13 at 14:44

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions must be relevant to professional system administration. Server Fault is a site dedicated to professionals; novice questions are off-topic. Please see the Help Center for more information on topicality. The best advice we can give you is to hire a professional to help you out." – Falcon Momot, Stephane
If this question can be reworded to fit the rules in the help center, please edit the question.

1 Answer 1

It seems to me that you're happy with rsnapshot but not with tar. If so, the solution is to replace tar with something that preserves the 'only changes' of rsnapshot.

Out of my head, I can thing of two easy options:

  1. after rsnapshot, find the differences between the last and previous snapshots to store only those to a tar file. It could be simply diff, with some flags tell it not to bother checking the content to provide the exact difference. The list of changed files should be enough. Another way would be to write some script to walk both snapshots and report new files and those with different inode numbers.

  2. use rdiff-backup, which uses the rsync algorithm but stores differences instead of transmitting them.

share|improve this answer
    
rdiff-backup gave me some ideas. I'm going to experiment a bit with that, ecryptfs and then rsync that to the remote server. First figuring out if rdiff-backup has all "features" I like, so it can replace rsnapshot. Or use both. –  Arlukin Sep 20 '13 at 21:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.