0

I have this folder, called it 'X'.

Currently, it has 40k folders and 70k files.

I am abit worried, inside are the images. Is there a limit that the folder 'X' can store?

That folder is important, it stores my current users images. Please advise?

Or should I store in database (BLOB)? I am using mysql.

Please advise.

EDIT:

I am using windows server 2003. IIS 6

1
  • 1
    @deadman: sorry, I had to remove your comment you've posted as answer on this page. The place for suggestions and discussions about the trilogy (SO, SF, SU) sites is meta.stackoverflow.com - thanks!
    – splattne
    Mar 24, 2010 at 12:15

6 Answers 6

3

It is all very well posting theoretical NTFS limits as some other posters have done, however you will find that in reality if you put 100k files in a folder, performance on accessing those files significantly drops. This is the case not just with NTFS but all the popular Linux filesystems as well.

The easiest method is to divide them into subfolders holding no more than 10k-20k each. If your naming is relatively evenly distributed you can do this by the first few characters of the filename to keep it simple. You want to create these folders in one go, not on the fly, so that the root folder is not fragmented.

There is also the issue of wasted space in blocks if you files are small and do not align to the block size. If this is the case then you may want to look at combining files to match it.

2
  • Agreed. Build a hierarchy Mar 24, 2010 at 12:46
  • +1, thank you for not blindly quoting a technical number like others have. Definitely keep a limit of less than 50k objects (files, folders, etc) per folder.
    – Chris S
    Mar 24, 2010 at 12:52
1

As far as I am aware, there is not a maximum limit to the amount of files in a folder, however NTFS has a a limit of 4,294,967,295 files on a volume.

2
  • Performance really sucks if you have more than 50k objects (files, foldets, and other things) in a folder.
    – Chris S
    Mar 24, 2010 at 12:50
  • 4,294,967,295 files is in laboratories. Windows sucks when files get close to 1000 in windows.
    – Alireza
    Jul 3, 2012 at 7:13
1

The Ext filesystem has at least performance problems if you have too many files in the same folder. I dont know for sure, but I would think that the same problems will arise with NTFS. It seams like a good idea to limit the number of files per folder to something reasonable by having a deeper folder tree.

"Something reasonable" has to be tested and measured in your environnement. I would go for below 10K, but that's just a hint ...

1

If you're using Windows 2003, then presumably your disks are NTFS formatted?

  • NTFS Maximum number of files on disk: 4,294,967,295
  • NTFS Maximum number of files in a single folder: 4,294,967,295

See here and here (scroll down to "NTFS Size Limits") for other relevant limitations of NTFS.

But this is one of those situations where if you're getting anywhere near the limits then you're almost certainly doing something wrong.

1

Those numbers are large, but not near the limits for the NTFS filesystem. So, while you won't get errors, you'll likely run into performance problems for example when retrieving directory listings.

In general about 6k-10k files in a directory will start to slow things down due to memory consumption. Above about 300k files, the creation of files will also slow down due to 'short file name generation' having trouble finding unique names.

Storing files in a MySQL blob is possible, but in my opinion it's slow and grows your database significantly. I would suggest creating a deeper directory structure, and migrating your current structure to it.

1
  • good point re: db blobs. It may seem like a good idea occasionally but everybody regrets it sooner or later if there are many and/or large files. Let dbs do what they are good at and let filesystems be used for what they are good at...if needed, manage the layout on the filesystem and reference those locations in the db.
    – damorg
    Mar 24, 2010 at 12:58
0

You didn't specify the file system or operating system. In Linux ext2 (and presumably ext3 and ext4, as they are based on it) has a theoretical limit of 1.3 × 10^20 directory entries.

1
  • @ptman: I am using Windows Server 2003. Any ideas?
    – deadman
    Mar 24, 2010 at 8:25

You must log in to answer this question.