Better folder structure for 1-6mil files?

Question

I'm doing a bit of development work for a start up, and have been asked to work on the directory structure for all the content being served and the scalability of the host servers(load balancing, etc).

At the moment theres about 500k files but its expected to continue to expand, each file is supposed to be unique but some are just older versions of the same file. All the files are going to be held in a sql db as well with more info on the files. Each file contains a tag that id's it, like file.coder.project Each file contains a tag that id's the revision, ex: 1 or 2 or 14 etc

So far the files have been in this structure(string is also stored in db):

 File\coder\project\file.coder.project.rev-md5.ext 

(file coder and project arent literally the segments, just an example)

Problem is some subfolders will be/are greater stuffed then others and I'm worried about replication issues accross multiple servers. I've been debating switching it to cutting up the value of its md5sum or sha to like 3/4 levels and just update the database(not a poblem, very easy)

The planned syncing process is going to be lsyncd and rsync scripts, since the db is goin to be replicated anyway.

Looking for other recommendations or ideas, or is md5/sha probably better for cutting up folder densities? Would either method effect file load/read times when accessed even though the full path would already be known?

All systems are going to be Ubuntu with either ext3 or ext4

Answer 1

The hash-based method of storing files has a lot going for it, but you do have to be sure to cut the hash up into enough chunks that the directories aren't too large. As I remember, for EXT3 directly opening a specific sub-directory in a directory filled with 15,000 sub-dirs takes longer than it would a directory with only 2,000 subdirs. Not sure about ext4.

Since hashes are significantly unique in their first few digits, carving the hash into 5 pieces, where the first 4 pieces are 3 characters of the hash and the last piece is larger then that, will keep that first level of directory under the 'very large' sizes. Directly accessing files in that kind of structure should be pretty fast with both EXT versions.