I work in bioinformatics and we store a lot of very large files which never change - plant genomes, genomic reads etc. We constantly receive new data of this type and the size of our backups is exploding.

In my opinion, it doesn't make sense to backup these large files all of the time, three or five times would be enough. Is there something like "stateful" backups which store what files are already "safe" (already on 5 tapes or so, using maybe file-hashes) and then only backup the rest?

I've googled around and found nothing.


  • If you only want to back a file up once, a differential backup should work. – Mark Henderson May 18 '12 at 1:08

Normally this is solved with either an incremental backup (backup all files since the last backup) or differential backup (all files since the last full backup). The Gnu Tar manual (section 5.2 and 5.3) has a brief discussion of these type of backups. However, this doesn't solve your problem of wanting a minimal number of copies of each file.

Another option, if you want to get an exact snapshot of the system on each backup, yet still save space is to use rsync snapshot backups (do a google search for rsync snapshot, there are several articles and tools that implement this). Basically this uses rsync to make copies to a remote system (or external drive), and uses hard links against files that don't change between each backup, to save space. To get multiple copies, you would then rsync your backup drive to another backup drive.

But, if you want this to all happen on tape, the only thing I'm aware of is commercial backup tools such at Tivoli. You might look into Bacula, which I think also supports keeping a minimum number of copies, but I haven't used that one yet.

Something that will be available soon, is a backup tool that I've been working on myself. I need to put together a bit more documentation and clean up the code before putting it up on github, but basically it does the snapshot-style incrementals-forever backups, keeping track of files by MD5 hash, and storing a catalog of snapshots of what a system looks like at each backup. It also, as a side effect, does file-level de-duplication when backing up multiple hosts to a single backup server. If you are interested, I'll come back later and update this post once I have the initial version of this tool uploaded (assuming it isn't against policy here to promote your own projects -- if it is, my apologies).

| improve this answer | |
  • Marking this as answer because it contains the other answers plus additional information - I'll most likely go to for the multiple-rsync-solution, but keep me posted about your project! It's unlikely that I convince my boss to go for that (as it's new and untried) but I'm still interested. Thanks to everyone! – Philipp May 19 '12 at 2:15
  • Derek, you might take a look at BackupPC – yaplik May 20 '12 at 12:28
  • @Philip, I created a project on code.google.com/snebu, which I will upload what I've got done so far. I just made some last minute improvements, so I want to put it through a validation test one last time before submitting it -- should have it in the next couple days. If you want, you can email me through the project page. – Derek Pressnall May 22 '12 at 3:59

Some variant of an incremental backup would work for this. Or perhaps you can set aside archived tapes containing static data on a regular basis to reduce the daily backup load.

| improve this answer | |
  • What if someone corrupted an archive and the incremental backup stores the corruption? But I agree we should set aside tapes with the usual static data.. – Philipp May 18 '12 at 1:21
  • @Philipp: If the data is static and not changed in any way, detecting corruptions is quite easy with checksums. – Sven May 18 '12 at 1:34
  • I'd say that the verification from your backup process should catch any corruption. Also you will be archiving to a different medium, correct? Are you using a backup software package? – ewwhite May 18 '12 at 1:45
  • Yes, we'll be archiving to tapes, and are currently using no software packages - just cp. I just started here and I want to change that. – Philipp May 18 '12 at 1:48

Store the files you import based on the date you get them. Hardlink them into the layout you need to use them in. Backup the last 5-7 days of directories.

| improve this answer | |

Store your data in different places and have different backup strategies. I have worked for a huge company and even there, that was the setup for the petabyte of data.

Something like:

  • /master for files that are almost immutable. A few user would usually upload big files to there. This was backuped once a month;
  • /data for all other files. there were links to /master files. This was backuped every night.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.