New answers tagged

-1

It's easy to design a hashing algorithm that is scalable over multiple cores, it's just that the best known hashing algorithms tend to be designed specifically to prevent that, in order that tasks like finding hash collisions are made as slow as possible. Hashing functions that don't force serial processing may suit you, but that depends what properties you ...


0

Most of the answers here have addressed the linear nature of most hashing algorithms. Although I'm sure there exists some true scalable hashing algorithms, an easier solution is to simply split the data up into smaller pieces, and hash each individually. Consider the BitTorrent approach: When a Torrent is created, all of the files are split into 'blocks', ...


-1

You can use md5deep for this and hashdeep for other hashes. It supports multi threading with the -j flag. By default it will create a hashing thread for each core. It also has a flag to break files into pieces before hashing but will not use multiple threads on a singe file. I've used this for getting sha256 of half a million files and it worked great. It ...


3

Unfortunately, MD5 is a linear process where its state depend on all the previous input. In other words, you can't truly parallelize it. Moreover, I'm not aware of any real hash alg that does not operate in this manner. What you can do (and, based on your answer, you are doing) is to split the source files and concurrently calculate each chunk's md5sum. If ...


6

Mine own best at the moment solution is: parallel --block=512M --pipepart -a …HUGEFILE… --progress --recend '' \ -k -j …NUMofProcessesSay4… md5sum | md5sum — It should be noted that: Resulting md5 hash isn't of the file, but rather of md5s of its parts but still it allows you to compare if replica is identical to origin It also doesn't perform very ...


2

There's pretty much no getting around processing the entire file. MD4 or CRC32 are probably your best bets for a widely deployed and fast algorithm (though CRC32 is going to be far less effective than MD4). Testing various implementation of your algorithm of choice will help. If you can find a well-tested asm implementation, it will likely improve upon the ...



Top 50 recent answers are included