Greetings,
I'm writing some scripts to process images from various photo websites. Right now I'm storing all this data in individual text files in the same directory.
The directory is web accessible. An end user makes a call to a web service which returns the path to the file the user will need.
I was wondering at what stage would I see a performance impact by having all these files in the same directory? (If any)
Performance varies according the the filesystem you're using.
EXT3: physical limit is 32,000 files, but perf suffers after several thousand files too.
EXT4: theoretically limitless
ReiserFS, XFS, JFS, BTRFS: these are the good ones for lots of files in a directory as they're more modern and designed to handle many files (the others were designed back in the days when HDDs were measured in MB not GB). Performance is a lot better for lots of files (along with ext4) as they both use a binary search type algorithm for getting the file you want (the others use a more linear one).
I store images for serving by a web server, and I have over 300,000 images in one directory on EXT3. I see no performance issues. Before setting this up, I did tests with 500k images in a directory, and randomly accessing files by name, and there was no significant slowdown with 500k over 10k images in the directory.
The only downside I see is that in order to sync the new ones with a second sever I have to run rsync over the whole directory, and can't just tell it to sync a sub directory containing the most recent thousand or so.
The amount of files in a folder could theoretically be limitless. However, every time when the OS will access the specific folder to search for files, it will have to process all files in the folder. With less than 500 files, you might not notice any delays. But when you have tens of thousands of files in a single folder, a simple folder list command (ls or dir) could take way too long. When these folders can be accessed through FTP, it will really be too slow...
Performance issues won't really depend on your OS but on your system processor speed, disk capacities and memory. If you have that many files, you might want to combine them into a single archive, and use an archiving system that is optimized to hold a lot of data. This could be a ZIP file but better yet, store them as blobs in a database with the file name as primary key.
ls is a problem.
Commented
Dec 31, 2009 at 10:33
My rule of thumb is to split folders if there are more than 1000 files and the folder will be browsed (i.e. through the internet or Explorer) or 5000 files otherwise.
As @skaffman points out, the limits depend on the operating system. You're likely to be affected by limits on older OSes. I remember an old version of Solaris was limited to 32768 files per directory.
The usual solution is to use some sort of hashing, i.e. Cyrus imap server splits users by an alphabetic hash:
/var/spool/imap/a/user/anna/
/var/spool/imap/a/user/albert/
/var/spool/imap/d/user/dan/
/var/spool/imap/e/user/ewan/
If you are directly accessing a file the number of files in a directory is no speed problem.
The number of files you can create in a single directory is depended on the file system you are using. If you are listing all files in the directory or searching, sorting, etc. having many files will slow down those operations.
gbjbaanb is wrong in his answer about the maximum file size of ext3. Generally ext limits the number of files on your disc in general. You can't create more files then you have inodes in your inode table. He is correct in suggesting reiserfs for more performance with many files
Checked folder with 10K files in NTFS (Windows 7, 64 bit). Folder with 10K images in any view (List, Icon etc.) works and scrolls without any sensible delay.
