How many subdirectories can a directory have on RH5 Linux?

Question

A program of mine is crunching numbers and depending on results creates a directory which stores all the data. Last night it failed. Mkdir no longer worked.

After checking things out I saw that my data directory has close to 36,000 sub directories underneath.

Deleting some of them helped fight the cause.

With that I ask you: is there a magic number of subdirectories that may exist underneath a parent directory? Does it have to do with the amount of data stored? (I don't think so, but still would like a confirmation)

In the above scenario, can I possibly reduce number of directories, maybe gzip some .. Will that solve the problem?

Generally under what circumstance will mkdir fail?

Thank you

Answer 1

Depends on which file system you're using. Reiserfs (my favorite) can handle up to 2^31 files per dir (that's 2 billion), with a max of 2^32 (4 billion) files on the filesys total. It can handle up to 64000 subdirs in a dir

Ext2 (and hence also ext3) has a limit of 32000 subdirs per dir. The max number of files per dir is theoretically unlimited (actually around 130 trillion), but performance becomes terrible with above 10-15 thousand files. The max number of total files on the filesys is limited by the number of inodes you have. With a 1 gig filesystem and a 4k block/inode ratio (the default), you have around 260000 inodes, and that's also the max number of files you can have.

How many files in a directory is too many? (Downloading data from net)

Answer 2

It depends on which filesystem you use. For ext3, there is a limit on 31998 subdirectories per directory. On ext3 it's also possible to run out of inodes before running out of free space if you have lots of very small files (the ratio of blocks per inode can be set a fs creation time).

More modern filesystems typically have higher limits, or are limited only by the available disk space.

That being said, that many directories or files is typically an indication that you're doing something wrong. Consider storing data in some binary scientific format such as netcdf, hdf5, or a database.

Answer 3

You have an ext4 file system. The max sub directories with ext4 is 64,000.

EDIT: It can technically go higher using the "dir_nlink" feature of ext4.