I'm doing a bit of development work for a start up, and have been asked to work on the directory structure for all the content being served and the scalability of the host servers(load balancing, etc).
At the moment theres about 500k files but its expected to continue to expand, each file is supposed to be unique but some are just older versions of the same file. All the files are going to be held in a sql db as well with more info on the files.
Each file contains a tag that id's it, like
Each file contains a tag that id's the revision, ex: 1 or 2 or 14 etc
So far the files have been in this structure(string is also stored in db):
(file coder and project arent literally the segments, just an example)
Problem is some subfolders will be/are greater stuffed then others and I'm worried about replication issues accross multiple servers. I've been debating switching it to cutting up the value of its md5sum or sha to like 3/4 levels and just update the database(not a poblem, very easy)
The planned syncing process is going to be lsyncd and rsync scripts, since the db is goin to be replicated anyway.
Looking for other recommendations or ideas, or is md5/sha probably better for cutting up folder densities? Would either method effect file load/read times when accessed even though the full path would already be known?
All systems are going to be Ubuntu with either ext3 or ext4