up vote 6 down vote favorite
1
share [g+] share [fb]

I'm curious, from a performance standpoint, is there an advantage in storing all files in one directory versus having each file in a separate directory? I'm not concerned about organization.

Also, this is assuming the files will be accessed often -- so I/O usage will be high. No directory listing is involved, the files will be pulled by absolute path.

The system environment is Linux, CentOS 5.3.

link|improve this question
feedback

4 Answers

up vote 5 down vote accepted

Path resolution is proportionate (though not linearly so) to the number of files in the directory. This is true even for resolving absolute paths because the file system still needs to scan the file names in each directory block to resolve the path. Different file systems have different resolution characteristics but, in general, you will start noticing the performance hit around 10,000 files.

link|improve this answer
feedback
up vote 0 down vote

Unless the directories are on different disks or RAIDs, then you won't see a noticeable difference, if they are all in one directory or not. The I/O operations for each disk are put together in one queue. If they are on different RAIDs, then you'll see the noticeable advantage you are looking for.

link|improve this answer
feedback
up vote 0 down vote

If you've got enough memory to hold all the files, have you considered caching them in RAM? http://www.linuxmaza.com/system-administration/how-to-mount-ramfs-tmpfs-in-linux/

link|improve this answer
1  
if they're so frequently accessed, they'll be cached by the OS – Javier Jul 2 '10 at 14:25
As long as you have enough memory free, Linux will cache your most recently used files in RAM, creating similar performance as with a ram disk. – tylerl Jul 3 '10 at 9:51
Yes, but it's possible that the contents of cache are emptied when it does something else, for instance, updatedb runs and sucks up memory for holding metadata. A ram disk guarantees that the memory is used for what you want – Matt Simmons Jul 3 '10 at 13:28
feedback
up vote 0 down vote

ext3 does some nice things:

http://www.ibm.com/developerworks/linux/library/l-fs8.html

See the section header: Journaling options and write latency
This allows you to "tune" ext3 for your application.

link|improve this answer
That's totally unrelated to this question, which is about directories. – Tobu Jul 2 '10 at 21:05
No - I'm referring to caching directories. I was not explicit I guess. – jim mcnamara Jul 3 '10 at 14:40
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.