Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two systems which hold backups and I would like to de-duplicate.

Both are existing systems so new filesystems such as zfs or lessfs aren't really an option.

There seems to be a HUGE list of deduplication software out there (example list stolen from wikipedia)...

  • duff
  • dupedit - Compares many files at once without checksumming. Avoids comparing files against themselves when multiple paths point to the same file.
  • dupmerge - runs on various platforms (Win32/64 with Cygwin, *nix, Linux etc.)
  • dupseek - Perl with algorithm optimized to reduce reads
  • fdf - Perl/c based and runs across most platforms (Win32, *nix and probably others). Uses MD5, SHA1 and other checksum algorithms
  • freedup - POSIX C compliant and runs across platforms (Windows with Cygwin, Linux, AIX, etc.)
  • freedups - shell script
  • fslint - has command line interface and GUI.
  • liten - Pure Python deduplication command line tool, and library, using md5 checksums and a novel byte comparison algorithm. (Linux, Mac OS X, *nix, Windows)
  • liten2 - A rewrite of the original Liten, still a command line tool but with a faster interactive mode using SHA-1 checksums (Linux, Mac OS X, *nix)
  • rdfind - One of the few which rank duplicates based on the order of input parameters (directories to scan) in order not to delete in "original/well known" sources (if multiple directories are given). Uses MD5 or SHA1.
  • rmlint - Fast finder with command line interface and many options to find other lint too (uses MD5)
  • ua - Unix/Linux command line tool, designed to work with [[find]] (and the like).
  • findrepe - free Java-based command-line tool designed for an efficient search of duplicate files, it can search within zips and jars.(GNU/Linux, Mac OS X, *nix, Windows)
  • fdupe - a small script written in Perl. Doing its job fast and efficiently.
  • ssdeep - identify almost identical files using Context Triggered Piecewise Hashing

... can anyone recommend one? Googling seems to throw up lots of stories of some programs taking 10 times longer than others.

Time isn't a huge issue as backup is all these machines do so I can nice it and leave it running for days but a 10x speed increase is big!

The main machine is a a Scientific Linux 6 (CentOS/RedHat) build that I can compile for but the other machine is a Netgear ReadyNAS which can only run python & perl scripts so bonus points for finding something that can run on that :-)

Thank you all in advance.

share|improve this question
1  
Unfortunately shopping and recommendation questions are off-topic for ServerFault‌​. Check out the FAQ. –  Brent Pabst Aug 15 '12 at 17:18
    
You've assembled a great list -- You now need to evaluate these options and see which works best for your environment. My advice however is to roll out a filesystem which supports deduplication (like ZFS) and migrate your data to it. Things that you graft onto your existing system will probably not perform as well as a filesystem/block level dedup system. –  voretaq7 Aug 15 '12 at 17:21
    
(If you can narrow down your list of choices and have questions about the differences between a small number (2-4) of options, or want to know how to do something specific with one of the tools we can definitely help with that...) –  voretaq7 Aug 15 '12 at 17:25
add comment

closed as not constructive by voretaq7 Aug 15 '12 at 17:20

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.