Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What are some good tools to Find duplicate files? Duplicate files are really bad when recording backup CDs/DVDs, then I want to pass a reliable duplicate file finder before the backup process.

I'm using Windows7, but any windows-compatible tool will be good.

share
    
What OS are you running? –  womble May 3 '09 at 22:28
comments disabled on deleted / locked posts

locked by Iain Aug 24 '12 at 12:45

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off topic by Iain Aug 24 '12 at 12:44

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

4 Answers

I like Ashisoft's Duplicate Finder on windows. The product is not free, but it has really nice methods for comparing files, it can use a byte-by-byte, checksum, ID3 tag information, or a few others. It has a selection tools to help you choose which files you want to keep from the dups.

share
add comment

I use Duplicate File Finder. It's available for 32-bit or 64-bit (x64) Windows NT/2000/XP, Windows 2003, and Windows Vista

share
add comment

Use Beyond Compare to visually compare files on your FTP server vs. a folder on your PC or simply to compare 2 local folders.

It can even compare the contents of many file types and visually show you the differences, for files such as text, pictures, MP3, and binary.

Its merging feature can help you consolidate scripts and text files.

share
1  
I love BC too, but it's not a dup file finder, it's a diff/merge tool. –  Scott Bilas Nov 9 '09 at 0:53
add comment

This is best handled in your backup software, if it's really that much of a problem (do you honestly have that many big duplicate files on your machine?) I know BackupPC purports to solve this problem by storing everything under the hash of it's contents and indexing from there, but whether that's compatible with your OS is unknown.

Finding dupes isn't hard -- just md5 hash every file on your system and anything with the same hash is a candidate for duplicate detection (still need to compare the actual file contents to make sure it's not a hash collision). 20 lines of shell script or Powershell should do it. But what are you going to do once you've found them? Deletion would work, if you only need one copy, but otherwise if you exclude them from the backup then the backup won't be complete. Far better to have the backup program deal with it for you.

share
4  
Doing md5 hashes of every file to look for duplicates is going to be frighteningly and horrifyingly slow. You would be much better off to start by comparing file sizes, and then doing md5 sums on files with matching sizes. Also, md5 collisions are a theoretical possibility, and yes there are some hacks out there that make use of that technique, but for the purpose of finding duplicate files the risk of collision is negligible. –  Christopher Cashell May 4 '09 at 22:07
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.