Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm looking for a quick way to compare directory contents. Is it possible to do an md5sum (or equivalent checksum) of an entire directory?

Using Ubuntu Linux

share|improve this question
    
You may also want to look into using diff to compare directories which will actually show you where the directories differ. unixtutorial.org/2008/06/how-to-compare-directories-in-unix –  Kibbee Aug 21 '11 at 2:38
    
@Kibbee To prevent that, you need to take into account something other than the data content of each file and exactly how you checksum the files. Given: [checksums] 1. A (Directory) - File1 [ABC] - File2 [CBA] 2. B (Directory) - File1 [ABC] - B1 (Directory) - File2 [CBA] 3. C (Directory) - File4 [ABC] - File5 [CBA] 4. D (Directory) - File1 copy [ABC] - File2 copy [CBA] Directory A and B are not identical although they contain the same files (although in B1, File2 is in a subdirectory). Under your example, A and C would be considered identical because –  Jacob Lyles Oct 19 '11 at 11:06
add comment

8 Answers

up vote 28 down vote accepted

Sure - md5sum directory/*

If you need something a little more flexible (say, for directory recursion or hash comparison), try md5deep.

apt-get install md5deep
md5deep -r directory

To compare a directory structure, you can give it a list of hashes to compare against:

md5deep -r -s /directory1 > dir1hashes
md5deep -r -X dir1hashes /directory2

This will output all of the files in directory2 that do not match to directory1.

share|improve this answer
    
Not what I meant, but what I wanted :) I did mean recursively, and getting ONE hash at the end, but I think this can be done with md5deep -l and hashing the output itself. –  pufferfish Aug 20 '11 at 17:25
1  
The order of the hashing is not consistent, so would have to sort the output before hashing –  pufferfish Aug 24 '11 at 12:13
1  
To get a deterministic order, use -j0 which disables multithreading (see the man page). –  Johann Mar 29 at 19:02
add comment

This worked for me: (run it while in the directory you are interested in)

md5deep -rl . | awk '{print $1}' | sort -n | md5sum
share|improve this answer
    
Great answer - thanks very much. –  Jasper Blues Mar 26 '13 at 13:19
    
Great, I really appreciate this. Thanks. –  user1149862 Apr 11 '13 at 8:34
add comment

The cfv application is quite useful, not only it can check and create MD5 checksums, it can also do CRC32, sha1, torrent, par, par2.

to create a CRC32 checksum file for all files in current directory:

cfv -C

to create a MD5 checksum file for all files in current directory:

cfv -C -t md5 -f "current directory.md5sums"

To create a separate checksum file for each sub directory:

cfv -C -r

To create a "super" checksum file containing files in all sub directories:

cfv -C -rr
share|improve this answer
add comment

I've had a need for verifying integrity of backups/mirrors which contain a large number of files and ended up writing a command-line program called MassHash. It's written in Python. A GTK+ Launcher is also available. You may want to check it out...

http://code.google.com/p/masshash/

share|improve this answer
add comment

As a specific case, lets say you want to copy some files from directory1 to directory2 and then you want to verify a successful copy using an md5 comparison.

First. cd to directory1 and type:

find -type f -exec md5sum "{}" \; > ~/Desktop/md5sum.txt

which will create a reference file containing an md5 sum for each file in directory1. Once this is done, all you have to do is cd to directory2 and type:

md5sum -c ~/Desktop/md5sum.txt

The program md5sum fetches each path from the md5sum.txt file, computes the md5sum of that file in the destination folder and then compares it with the sum it has stored in the file.

After the process is complete, you will get a summary such as 'So and so many files didn't match up' or something like that.

share|improve this answer
1  
Reference: ubuntugenius.wordpress.com/2009/11/17/… –  Joel Feb 12 '12 at 12:05
add comment

i think i answered this one before with this answer:

find . -xtype f -print0 | xargs -0 sha1sum | cut -b-40 | sort | sha1sum

gives: b1a5b654afee985d5daccd42d41e19b2877d66b1

the idea is you hash all the files cut out the hashes one per line, sort them and hash that yielding a single hash. this doesn't depend on the names of the files.

share|improve this answer
add comment

You could create MD5 sums of every single files, order these checksums alphabetically and has them (with or without newlines). Since MD5 is cryptographic, it should work just fine with hashes of hashes.

There should be a certain order to things, otherwise you will get different results for equal dirs.

And you should consider that adding some file to one dir will completely change the result, even if it was just a .directory of .DS_Store file.

share|improve this answer
    
Technically one could get the same hash for different directories. If dir A had 2 files with contents 'ab' and 'c' and dir B had 2 files with contents 'a' and 'bc' then hashing only the data in the files would yield the same results, even though they have files with different contents. I'm not even sure how one would define the MD5Sum of a directory. –  Kibbee Aug 21 '11 at 2:37
add comment

If you'd like to see what's different (if anything) between two directories, rsync would be a good fit.

rsync --archive --dry-run --checksum --verbose /source/directory/ /destination/directory

This will list any files that are different.

share|improve this answer
1  
diff -qr /source/directory/ /destination/directory/ would also show files that differ. –  Konerak Aug 21 '11 at 14:36
    
Is there a way to perform a bitwise comparison instead of checksums? It might be faster on local drives. –  Ali Aug 21 '11 at 15:03
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.