I'm writing some scripts to process images from various photo websites. Right now I'm storing all this data in individual text files in the same directory.

The directory is web accessible. An end user makes a call to a web service which returns the path to the file the user will need.

I was wondering at what stage would I see a performance impact by having all these files in the same directory? (If any)


7 Answers 7


Performance varies according the the filesystem you're using.

  • FAT: forget it :) (ok, I think the limit is 512 files per directory)
  • NTFS: Althought it can hold 4billion files per folder, it degrades relatively quickly - around a thousand you will start to notice performance issues, several thousand and you'll see explorer appear to hang for quite a while.
  • EXT3: physical limit is 32,000 files, but perf suffers after several thousand files too.

  • EXT4: theoretically limitless

  • ReiserFS, XFS, JFS, BTRFS: these are the good ones for lots of files in a directory as they're more modern and designed to handle many files (the others were designed back in the days when HDDs were measured in MB not GB). Performance is a lot better for lots of files (along with ext4) as they both use a binary search type algorithm for getting the file you want (the others use a more linear one).

  • 7
    This is wrong. There isn't a limit of 32000 files in EXT3. There's a limit of 32000 subdirectories. I've got a directory here with over 300000 files and it performs fine. Dec 31, 2009 at 10:39
  • 1
    quite true - the file limit is the entire filesystem's limit on inodes, but you're limited to 32k links (ie subdirs).
    – gbjbaanb
    Jan 1, 2010 at 17:16
  • The statement for current NTFS is also not true, it can hold up to 4,294,967,295 (2^32 - 1): technet.microsoft.com/en-us/library/cc781134%28WS.10%29.aspx Jun 18, 2013 at 16:22
  • Do NOT confuse sub-directories with files, on CentOS machine I had 32000 sub-directories, reached the limit, I moved all FILES in that one directory and still works fine.
    – adrianTNT
    Aug 1, 2013 at 19:04
  • Some numbers for MacOS here
    – smci
    Nov 20, 2017 at 23:27

I store images for serving by a web server, and I have over 300,000 images in one directory on EXT3. I see no performance issues. Before setting this up, I did tests with 500k images in a directory, and randomly accessing files by name, and there was no significant slowdown with 500k over 10k images in the directory.

The only downside I see is that in order to sync the new ones with a second sever I have to run rsync over the whole directory, and can't just tell it to sync a sub directory containing the most recent thousand or so.

  • Well, to synic with a second server i think you must create a structure and algorithm that keeps the changes, then this log can save you a great amount of time. Mar 13, 2016 at 13:50
  • 1
    +1 This actually answers the question.
    – kubanczyk
    Mar 18, 2016 at 10:51
  • One downside, if you use an FTP client like FileZilla and want to list the content of the folder, it takes a while.
    – Avatar
    Jun 21, 2018 at 16:25

The amount of files in a folder could theoretically be limitless. However, every time when the OS will access the specific folder to search for files, it will have to process all files in the folder. With less than 500 files, you might not notice any delays. But when you have tens of thousands of files in a single folder, a simple folder list command (ls or dir) could take way too long. When these folders can be accessed through FTP, it will really be too slow...

Performance issues won't really depend on your OS but on your system processor speed, disk capacities and memory. If you have that many files, you might want to combine them into a single archive, and use an archiving system that is optimized to hold a lot of data. This could be a ZIP file but better yet, store them as blobs in a database with the file name as primary key.

  • But will accessing the file directly remove bottlenecks with searching directories or will accessing a directy still have an underlying search call ? (Linux, debian)
    – steve
    Dec 30, 2009 at 14:49
  • 3
    Accessing the file directly will mitigate these problems. I've done tests on ext3, and accessing a file by name in a directory containing 500000 files is not significantly slower than one containing 1000. Obviously doing an ls is a problem. Dec 31, 2009 at 10:33
  • When knowing the exact name, access should be fast. The problem would be mostly any code or command that wants to get a list of files. Jan 1, 2010 at 1:09

My rule of thumb is to split folders if there are more than 1000 files and the folder will be browsed (i.e. through the internet or Explorer) or 5000 files otherwise.


As @skaffman points out, the limits depend on the operating system. You're likely to be affected by limits on older OSes. I remember an old version of Solaris was limited to 32768 files per directory.

The usual solution is to use some sort of hashing, i.e. Cyrus imap server splits users by an alphabetic hash:

  • 1
    Thanks, I'd definitly have something in place once a dir has more than 2k files! :)
    – steve
    Dec 30, 2009 at 14:48
  • This question has some good answers: serverfault.com/questions/95444/…
    – gm3dmo
    Dec 30, 2009 at 15:26
  • My general rule of thumb is that more than about 20,000 files in a directory is not a good idea. Most modern filesystems do ok with that many files. Once you hit 32k files in a directory some filesystems such as ext3 will start having serious performance issues. Dec 30, 2009 at 20:17
  • Phil - do you have any information on the performance issues with over 32k files with ext3, I'm not seeing any at the moment with over 300k Maybe it's something that isn't affecting my pattern of use. Dec 31, 2009 at 10:41
  • At my previous job scientific software would generate lots of small (few k each) files in a directory. We definitely saw that for >32k files directory read times would shoot up hugely. Just running 'ls' on a directory with that many files would take a minute or more. Jan 7, 2010 at 5:43

If you are directly accessing a file the number of files in a directory is no speed problem.

The number of files you can create in a single directory is depended on the file system you are using. If you are listing all files in the directory or searching, sorting, etc. having many files will slow down those operations.

gbjbaanb is wrong in his answer about the maximum file size of ext3. Generally ext limits the number of files on your disc in general. You can't create more files then you have inodes in your inode table. He is correct in suggesting reiserfs for more performance with many files


Checked folder with 10K files in NTFS (Windows 7, 64 bit). Folder with 10K images in any view (List, Icon etc.) works and scrolls without any sensible delay.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.