Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

OK, not so large but I need to use something where around 60,000 files with average size of 30kb are stored in a single directory (this is a requirement so can't simply break into sub-directories with smaller number of files).

The files will be accessed randomly, but once created there will be no writes to the same filesystem. I'm currently using Ext3 but finding it very slow. Any suggestions?

share|improve this question
2  
Why must they be in one directory? –  Kyle Brandt Jul 20 '09 at 19:48
    
I am also interested in an up to date answer to original question, given enough improvements in xfs and ext4. –  user127752 Jul 11 '12 at 17:40

11 Answers 11

You should consider XFS. It supports a very large number of files both at the filesystem and at the directory level, and the performance remains relatively consistent even with a large number of entries due to the B+ tree data structures.

There's a page on their wiki to a large number of papers and publications that detail the design. I recommend you give it a try and benchmark it against your current solution.

share|improve this answer
    
according to the slides in @nelaar's answer, ext4 would be superior to xfs for this task. –  mulllhausen Jul 16 at 14:13

One billion files on Linux

The author of this article digs into some of the performance issues on file systems with large files counts and does some nice comparisons of the performance of various file systems ext3, ext4 and XFS. This is made available as a slide show. http://events.linuxfoundation.org/slides/2010/linuxcon2010_wheeler.pdf

time to run mkfs time to create 1M 50kb files File system repair time removing 1m files

share|improve this answer
1  
We really do prefer that answers contain content not pointers to content. Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. –  Iain Aug 27 '12 at 11:55
    
@Iain I hope that is better, as simply downloading the PDF, would give you the same info. –  nelaar Aug 27 '12 at 12:25

Many files in a directory on ext3 has been discussed in length at the sister site stackoverflow.com

In my opinion 60 000 files in one directory on ext3 is far from ideal but depending on your other requirements it might be good enough.

share|improve this answer

OK. I did some preliminary testing using ReiserFS, XFS, JFS, Ext3 (dir_hash enabled) and Ext4dev (2.6.26 kernel). My first impression was that all were fast enough (on my beefy workstation) - it turns out that the remote production machine has a fairly slow processor.

I experienced some weirdness with ReiserFS even on initial testing so ruled that out. It seems that JFS has 33% less CPU requirement than all the others and so will test that out on the remote server. If it performs well enough, I'll use that.

share|improve this answer

Using tune2fs to enable dir_index might help. To see if it is enabled:

sudo tune2fs -l /dev/sda1 | grep dir_index

If it is not enabled:

sudo umount /dev/sda1   
sudo tune2fs -O dir_index /dev/sad1
sudo e2fsck -D /dev/sda1
sudo mount /dev/sda1

But I have a feeling you might be going down the wrong path... why not generate a flat index and use some code to select randomly based on that. You can then use sub directories for a more optimized tree structure.

share|improve this answer
    
Thanks. dir_index was enabled. –  bugmenot77 Jul 20 '09 at 22:10

I'm writing an application that also stores lots and lots of files although mine are bigger and I have 10 million of them that I'll be splitting across multiple directories.

ext3 is slow mainly because of the default "linked list" implementation. So if you have lots of files in one directory, it means opening or creating another is going to get slower and slower. There is something called an htree index that is available for ext3 that reportedly improves things greatly. But, it's only available on filesystem creation. See here: http://lonesysadmin.net/2007/08/17/use-dir_index-for-your-new-ext3-filesystems/

Since you're going to have to rebuild the filesystem anyway and due to the ext3 limitations, my recommendation is that you look at using ext4 (or XFS). I think ext4 is a little faster with smaller files and has quicker rebuilds. Htree index is default on ext4 as far as I'm aware. I don't really have any experience with JFS or Reiser but I have heard people recommend that before.

In reality, I'd probably test several filesystems. Why not try ext4, xfs & jfs and see which one gives the best overall performance?

Something that a developer told me that can speed things up in the application code is not to do a "stat + open" call but rather "open + fstat". The first is significantly slower than the second. Not sure if you have any control or influence over that.

See my post here on stackoverflow. Storing & accessing up to 10 million files in Linux there are some very useful answers and links there.

share|improve this answer

You can store file inodes instead of filenames: accessing inode numbers should be much faster that resolving file names

share|improve this answer
    
Now tell me. How do you open a file by inode number? –  Matt Feb 24 '11 at 1:38
1  
@Matt, It looks like the question has changed after I answered. Or I was much more stupid a 1.5 years ago :))) –  kolypto Feb 24 '11 at 21:26

ext3 and below support up to 32768 files per directory. ext4 supports up to 65536 in the actual count of files, but will allow you to have more (it just won't store them in the directory, which doesn't matter for most user purposes).

Also, the way directories are stored on ext* filesystems is essentially as one big list. On the more modern filesystems (Reiser, XFS, JFS) they are stored as B-trees, which are much more efficient for large sets.

share|improve this answer
1  
supporting that number of files in a dir is not the same thing as doing it at a reasonable speed. i don't know yet whether ext4 is any better, but ext3 slows down greatly when it has more than a few thousand files in a directory, even with dir_index turned on (it helps, but doesn't eliminate the problem entirely). –  cas Jul 20 '09 at 21:50

You dont want to cram that many files in one directory, you want some sort of structure. Even if it's something as simple as haveing subdirectories that start with the first character of the file can improve your access times. Another silly trick I like to use, is to force the system to update it's cache with metainformation is to run updatedb regularly. In one window run slabtop, and in another run updatedb and you'll see much memory is going to get allocated to caching. It's much faster this way.

share|improve this answer

You didn't specify the kind of data in these files. But from the sounds of it, you should be using some sort of database with indexing for quick searches.

share|improve this answer

Filesystem is probably not the ideal storage for such requirement. Some kind of database storage is better. Still if you cant help it, then try splitting files in several directories and use unionfs to mount(bind) those directories on single directory where you want all files to appear. I have not used this technique for speed up at all, but it worth a try.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.