Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm doing a bit of development work for a start up, and have been asked to work on the directory structure for all the content being served and the scalability of the host servers(load balancing, etc).

At the moment theres about 500k files but its expected to continue to expand, each file is supposed to be unique but some are just older versions of the same file. All the files are going to be held in a sql db as well with more info on the files. Each file contains a tag that id's it, like file.coder.project Each file contains a tag that id's the revision, ex: 1 or 2 or 14 etc

So far the files have been in this structure(string is also stored in db):

 File\coder\project\file.coder.project.rev-md5.ext 

(file coder and project arent literally the segments, just an example)

Problem is some subfolders will be/are greater stuffed then others and I'm worried about replication issues accross multiple servers. I've been debating switching it to cutting up the value of its md5sum or sha to like 3/4 levels and just update the database(not a poblem, very easy)

The planned syncing process is going to be lsyncd and rsync scripts, since the db is goin to be replicated anyway.

Looking for other recommendations or ideas, or is md5/sha probably better for cutting up folder densities? Would either method effect file load/read times when accessed even though the full path would already be known?

All systems are going to be Ubuntu with either ext3 or ext4

share|improve this question

migrated from superuser.com Oct 28 '11 at 11:39

This question came from our site for computer enthusiasts and power users.

1  
rsync is not going to perform well for a large number of files. You may want to look at some block level replication, or distributed filesystem. –  Zoredache Oct 27 '11 at 19:40
1  
There's a lot of factors that go into this decision, but have you thought about storing the files as blobs inside the database? I'm normally not a fan of that kind of plan, but it's looking like there might be a good case for it here. –  afrazier Oct 27 '11 at 20:00
add comment

1 Answer 1

The hash-based method of storing files has a lot going for it, but you do have to be sure to cut the hash up into enough chunks that the directories aren't too large. As I remember, for EXT3 directly opening a specific sub-directory in a directory filled with 15,000 sub-dirs takes longer than it would a directory with only 2,000 subdirs. Not sure about ext4.

Since hashes are significantly unique in their first few digits, carving the hash into 5 pieces, where the first 4 pieces are 3 characters of the hash and the last piece is larger then that, will keep that first level of directory under the 'very large' sizes. Directly accessing files in that kind of structure should be pretty fast with both EXT versions.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.