I know this is old, but extremely relevant as duplicating 100's of GB's to Tb's of uneeded files often is problematic for anyone budget conscious.
I have been using Areca Backup. and it seems to keep track of duplicate files well. I recently moved 300GB one day and it only backup up 8GB actual data meaning it did not recopy the files and just referenced them
My manual file mirror, versions, and deletions had bloated 350GB (300 noted above) in 3 months where ArecaBackup only bloated 20GB over the same time frame.
I appreciate it's ability to backup the files in such a way that you can access the actual file in a directory tree on the backend; that is files can be stored in a directory tree in their original format instead of a proprietary format to fight possibility of corruption of a backup. Although typically you would browse through the GUI.
- It is free and open source GPL2.
- It works.
- It can output a script so you can easily run a cron job around it.
- It supports full, differntial, and incremental backups
- It supports compression
- It has a Delta transfer option.
- It has a GUI which is very nice for recovery
- It has a plethora of options and customization you can do as well.
- It is java based so it is a bit platform agnostic
I have not found a way to delete a file from the archive, so if there is a large file I want gone I delete it in the directory structure and if someone tried to recover it for some reason I'm sure it would throw an error when recovering it. Although I only do this for known large temporary or duplicate files so this has not been an issue; namely from users using backed-up locations as a scratch space and the scratch work ends up getting backed up.
The biggest danger is if your configuration files get corrupted or go missing you cannot rebuild them; but you still would have your files.
Overall if you do not like vendor lock in or proprietary file formats, it is a great solution.
I have no affiliation with them, they just provided something that solves a problem for me!