(HDD) backup solution in a space-constrained environment [closed]

Question

I am looking for a backup software (FOSS, running on Linux) which is able to back up about 1 TiB of data in mostly big files (smallest around 3 MiB, largest around 4 GiB) to a hard disk drive.

The clou is: On the backup hard disk drive, there needs to be at least one working copy of the data at any point in time. However, I do not want to store two full (!) copies of the data, not even temporarily. Therefore, let’s assume I only have 1.5 TiB capacity on my backup hard disk.

In order for the backup software to figure out which files were just saved twice and delete the old copy, it needs to operate on file level (sub-file level is not required). However, incremental backups are not necessary — I always do full backups. The motivation behind always doing full backups is to eliminate the possibility of bitrot and detect hard disk drive failures early by keeping them busy and rewriting every sector once in a while.

Which backup software fulfills my requirements? I looked at rsnapshot, bacula, duplicity, obnam and bup. To the best of my knowledge, none of them fulfills this requirement.

Answer 1

For backing up large files the best approach I found is using de-duplication. I'm currently not aware of any OSS that uses deduplication but you could play with SDFS from Opendedup project and combine it with tools you mentioned. I personally use Asigra software, which is probably an overkill for your situation.

Answer 2

There are at least few solutions that either use de-duplication or copy-on-write snapshots:

1. rdiff-backup: will handle large number of files easily, will provide clear-view of last snapshot. very space efficient for files that change only a little
2. lessfs: you'll need to copy files around as current stable version doesn't support snapshots, but it's a de-duplicating FS so it won't take much space and snapshots are in development
3. btrfs, while not generally regarded as proven and completely stable, Oracle will be releasing Unbreakable Linux kernel for their Enterprise Linux with official support for btrfs (14.02.2012)