Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I work in bioinformatics and we store a lot of very large files which never change - plant genomes, genomic reads etc. We constantly receive new data of this type and the size of our backups is exploding.

In my opinion, it doesn't make sense to backup these large files all of the time, three or five times would be enough. Is there something like "stateful" backups which store what files are already "safe" (already on 5 tapes or so, using maybe file-hashes) and then only backup the rest?

I've googled around and found nothing.

Thanks!

share|improve this question
    
If you only want to back a file up once, a differential backup should work. –  Mark Henderson May 18 '12 at 1:08
add comment

4 Answers

up vote 3 down vote accepted

Normally this is solved with either an incremental backup (backup all files since the last backup) or differential backup (all files since the last full backup). The Gnu Tar manual (section 5.2 and 5.3) has a brief discussion of these type of backups. However, this doesn't solve your problem of wanting a minimal number of copies of each file.

Another option, if you want to get an exact snapshot of the system on each backup, yet still save space is to use rsync snapshot backups (do a google search for rsync snapshot, there are several articles and tools that implement this). Basically this uses rsync to make copies to a remote system (or external drive), and uses hard links against files that don't change between each backup, to save space. To get multiple copies, you would then rsync your backup drive to another backup drive.

But, if you want this to all happen on tape, the only thing I'm aware of is commercial backup tools such at Tivoli. You might look into Bacula, which I think also supports keeping a minimum number of copies, but I haven't used that one yet.

Something that will be available soon, is a backup tool that I've been working on myself. I need to put together a bit more documentation and clean up the code before putting it up on github, but basically it does the snapshot-style incrementals-forever backups, keeping track of files by MD5 hash, and storing a catalog of snapshots of what a system looks like at each backup. It also, as a side effect, does file-level de-duplication when backing up multiple hosts to a single backup server. If you are interested, I'll come back later and update this post once I have the initial version of this tool uploaded (assuming it isn't against policy here to promote your own projects -- if it is, my apologies).

share|improve this answer
    
Marking this as answer because it contains the other answers plus additional information - I'll most likely go to for the multiple-rsync-solution, but keep me posted about your project! It's unlikely that I convince my boss to go for that (as it's new and untried) but I'm still interested. Thanks to everyone! –  Philipp May 19 '12 at 2:15
    
Derek, you might take a look at BackupPC –  yaplik May 20 '12 at 12:28
    
@Philip, I created a project on code.google.com/snebu, which I will upload what I've got done so far. I just made some last minute improvements, so I want to put it through a validation test one last time before submitting it -- should have it in the next couple days. If you want, you can email me through the project page. –  Derek Pressnall May 22 '12 at 3:59
    
Cool! Thank you very much, I'll have a look –  Philipp May 23 '12 at 4:55
add comment

Some variant of an incremental backup would work for this. Or perhaps you can set aside archived tapes containing static data on a regular basis to reduce the daily backup load.

share|improve this answer
    
What if someone corrupted an archive and the incremental backup stores the corruption? But I agree we should set aside tapes with the usual static data.. –  Philipp May 18 '12 at 1:21
    
@Philipp: If the data is static and not changed in any way, detecting corruptions is quite easy with checksums. –  SvW May 18 '12 at 1:34
    
I'd say that the verification from your backup process should catch any corruption. Also you will be archiving to a different medium, correct? Are you using a backup software package? –  ewwhite May 18 '12 at 1:45
    
Yes, we'll be archiving to tapes, and are currently using no software packages - just cp. I just started here and I want to change that. –  Philipp May 18 '12 at 1:48
add comment

Store the files you import based on the date you get them. Hardlink them into the layout you need to use them in. Backup the last 5-7 days of directories.

share|improve this answer
add comment

Store your data in different places and have different backup strategies. I have worked for a huge company and even there, that was the setup for the petabyte of data.

Something like:

  • /master for files that are almost immutable. A few user would usually upload big files to there. This was backuped once a month;
  • /data for all other files. there were links to /master files. This was backuped every night.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.