up vote 4 down vote favorite
2
share [g+] share [fb]

What are some good tools to Find duplicate files? Duplicate files are really bad when recording backup CDs/DVDs, then I want to pass a reliable duplicate file finder before the backup process.

I'm using Windows7, but any windows-compatible tool will be good.

link|improve this question
54% accept rate
What OS are you running? – womble May 3 '09 at 22:28
feedback

4 Answers

up vote 2 down vote

I like Ashisoft's Duplicate Finder on windows. The product is not free, but it has really nice methods for comparing files, it can use a byte-by-byte, checksum, ID3 tag information, or a few others. It has a selection tools to help you choose which files you want to keep from the dups.

link|improve this answer
feedback
up vote 1 down vote

I use Duplicate File Finder. It's available for 32-bit or 64-bit (x64) Windows NT/2000/XP, Windows 2003, and Windows Vista

link|improve this answer
feedback
up vote 1 down vote

Use Beyond Compare to visually compare files on your FTP server vs. a folder on your PC or simply to compare 2 local folders.

It can even compare the contents of many file types and visually show you the differences, for files such as text, pictures, MP3, and binary.

Its merging feature can help you consolidate scripts and text files.

link|improve this answer
1  
I love BC too, but it's not a dup file finder, it's a diff/merge tool. – Scott Bilas Nov 9 '09 at 0:53
feedback
up vote 0 down vote

This is best handled in your backup software, if it's really that much of a problem (do you honestly have that many big duplicate files on your machine?) I know BackupPC purports to solve this problem by storing everything under the hash of it's contents and indexing from there, but whether that's compatible with your OS is unknown.

Finding dupes isn't hard -- just md5 hash every file on your system and anything with the same hash is a candidate for duplicate detection (still need to compare the actual file contents to make sure it's not a hash collision). 20 lines of shell script or Powershell should do it. But what are you going to do once you've found them? Deletion would work, if you only need one copy, but otherwise if you exclude them from the backup then the backup won't be complete. Far better to have the backup program deal with it for you.

link|improve this answer
4  
Doing md5 hashes of every file to look for duplicates is going to be frighteningly and horrifyingly slow. You would be much better off to start by comparing file sizes, and then doing md5 sums on files with matching sizes. Also, md5 collisions are a theoretical possibility, and yes there are some hacks out there that make use of that technique, but for the purpose of finding duplicate files the risk of collision is negligible. – Christopher Cashell May 4 '09 at 22:07
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.