Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 0 down vote favorite

I am looking for a backup software (FOSS, running on Linux) which is able to back up about 1 TiB of data in mostly big files (smallest around 3 MiB, largest around 4 GiB) to a hard disk drive.

The clou is: On the backup hard disk drive, there needs to be at least one working copy of the data at any point in time. However, I do not want to store two full (!) copies of the data, not even temporarily. Therefore, let’s assume I only have 1.5 TiB capacity on my backup hard disk.

In order for the backup software to figure out which files were just saved twice and delete the old copy, it needs to operate on file level (sub-file level is not required). However, incremental backups are not necessary — I always do full backups. The motivation behind always doing full backups is to eliminate the possibility of bitrot and detect hard disk drive failures early by keeping them busy and rewriting every sector once in a while.

Which backup software fulfills my requirements? I looked at rsnapshot, bacula, duplicity, obnam and bup. To the best of my knowledge, none of them fulfills this requirement.

share|improve this question

closed as too localized by SvW, John Gardeniers, EEAA, Ward, MDMarra Feb 13 '12 at 20:20

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, see the FAQ.

2 Answers

up vote 0 down vote

For backing up large files the best approach I found is using de-duplication. I'm currently not aware of any OSS that uses deduplication but you could play with SDFS from Opendedup project and combine it with tools you mentioned. I personally use Asigra software, which is probably an overkill for your situation.

share|improve this answer
up vote 0 down vote

There are at least few solutions that either use de-duplication or copy-on-write snapshots:

  1. rdiff-backup: will handle large number of files easily, will provide clear-view of last snapshot. very space efficient for files that change only a little
  2. lessfs: you'll need to copy files around as current stable version doesn't support snapshots, but it's a de-duplicating FS so it won't take much space and snapshots are in development
  3. btrfs, while not generally regarded as proven and completely stable, Oracle will be releasing Unbreakable Linux kernel for their Enterprise Linux with official support for btrfs (14.02.2012)
share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.