Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

As far as I know, the time complexity of scanning a directory and the amount of files in said directory are correlated due to I/O costs. Would the administrative costs of placing the files in a hashed directory tree for uploading/downloading files through a CDN API be worth it for the added efficiency?

For instance, given a filename foo.mp3, the MD5 hash for this is 10ebb1120767e9de166e0f5905077cb1. Thus, storing foo.mp3 in ./10/eb/foo.mp3 would allow for less files per directory (assuming MD5 generates patterns with in Base36, this allows for 36^2 root directories with 36^2 subdirectories each and little chance of hash collision)

Considering the directories themselves are not loaded, would the I/O costs of directory scanning still exist with direct uploading/downloading?

share|improve this question
1  
Are you asking whether it would be more efficient to put each file in it's own directory, instead of all the files in one directory? –  John K Aug 17 '12 at 15:22
    
Question updated. –  Daniel Li Aug 17 '12 at 15:26

1 Answer 1

up vote 1 down vote accepted

In theory such a technique would greatly reduce bandwidth costs, however it would also increase CPU costs. I suggest you have a look at Rsync and see how this tool has solved this problem without the use of hash directories. Also given specific data patters it is possible to have hash collisions where two files would have the exact same MD5 hash. This only becomes a problem if the new and old version of a specific file have the same MD5 hash. It is rare, but it does happen.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.