Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm curious, from a performance standpoint, is there an advantage in storing all files in one directory versus having each file in a separate directory? I'm not concerned about organization.

Also, this is assuming the files will be accessed often -- so I/O usage will be high. No directory listing is involved, the files will be pulled by absolute path.

The system environment is Linux, CentOS 5.3.

share|improve this question

4 Answers 4

up vote 6 down vote accepted

Path resolution is proportionate (though not linearly so) to the number of files in the directory. This is true even for resolving absolute paths because the file system still needs to scan the file names in each directory block to resolve the path. Different file systems have different resolution characteristics but, in general, you will start noticing the performance hit around 10,000 files.

share|improve this answer

Unless the directories are on different disks or RAIDs, then you won't see a noticeable difference, if they are all in one directory or not. The I/O operations for each disk are put together in one queue. If they are on different RAIDs, then you'll see the noticeable advantage you are looking for.

share|improve this answer

If you've got enough memory to hold all the files, have you considered caching them in RAM? http://www.linuxmaza.com/system-administration/how-to-mount-ramfs-tmpfs-in-linux/

share|improve this answer
1  
if they're so frequently accessed, they'll be cached by the OS –  Javier Jul 2 '10 at 14:25
    
As long as you have enough memory free, Linux will cache your most recently used files in RAM, creating similar performance as with a ram disk. –  tylerl Jul 3 '10 at 9:51
    
Yes, but it's possible that the contents of cache are emptied when it does something else, for instance, updatedb runs and sucks up memory for holding metadata. A ram disk guarantees that the memory is used for what you want –  Matt Simmons Jul 3 '10 at 13:28

ext3 does some nice things:

http://www.ibm.com/developerworks/linux/library/l-fs8.html

See the section header: Journaling options and write latency
This allows you to "tune" ext3 for your application.

share|improve this answer
    
That's totally unrelated to this question, which is about directories. –  Tobu Jul 2 '10 at 21:05
    
No - I'm referring to caching directories. I was not explicit I guess. –  jim mcnamara Jul 3 '10 at 14:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.