Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In a social networking project we want to store user's avatars in a folder. I think in one year or two it'll reach to 140K (I've seen this issue before and it will be around this number). I want to spread files in folders. If a folder contains 1000 files then create another folder and do store files from 1001 to 2000. Is this a good approach or I'm just very cautious about the issue? (File system : EXT3)

share|improve this question
1  
This doesn't really answer your question so I'll just post it as a comment. I'd seriously consider using something like MogileFS for this sort of thing, and not have to worry about the various quirks of filesystems. –  ThatGraemeGuy Jul 3 '12 at 9:47

2 Answers 2

up vote 2 down vote accepted

Over time ext3 is not a stellar choice for directories containing huge number of files, even if it has gained directory indexes and other features. This is true especially if the directory has lots of write activity in addition to frequent read activity, or if files in that directory are served over NFS/Samba.

In general it would be better design to hash the files in subdirectories somehow, such as

a/apple.png
b/banana.png
c/cow.png

or

a/ap/apple.png
b/ba/banana.png
c/co/cow.png

The suggestion you made (after some number of files in one directory create an another directory) is also good.

share|improve this answer
    
never thought about it this way. I think it is better to store files by letters, but there maybe some problem with it like a folder gets bloated and other folders empty. –  Areza Jul 3 '12 at 7:38
1  
علیرضا_ Sure, there will be slight inbalance between directories unless you are really lucky. But unless almost all your file names start with the same letter, my solution should be Good Enough(tm). –  Janne Pikkarainen Jul 3 '12 at 7:41
    
I've got my answer thanks. last thing: If you were me how many files would you put in that directory before switching to another. –  Areza Jul 3 '12 at 7:48
1  
علیرضا: Test it for your workload. Anything from couple of thousands to around 10 000 should be OK, don't go too much over that. –  Janne Pikkarainen Jul 3 '12 at 7:50

See Directory Indexing in Features Found in Linux 2.6. It mentions directories with over 16 million files. Should be enough for a while.

share|improve this answer
2  
Yes, that is the theoretical limit. In practice, nasty things happen after enough time and write modification to that directory. The directory entry itself becomes fragmented and leads to unnecessary disk seeks & slow performance. –  Janne Pikkarainen Jul 3 '12 at 7:38
    
@JannePikkarainen couldn't agree more. These laboratory tests is not really good in practice. –  Areza Jul 3 '12 at 7:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.