Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

A program of mine is crunching numbers and depending on results creates a directory which stores all the data. Last night it failed. Mkdir no longer worked.

After checking things out I saw that my data directory has close to 36,000 sub directories underneath.

Deleting some of them helped fight the cause.

With that I ask you: is there a magic number of subdirectories that may exist underneath a parent directory? Does it have to do with the amount of data stored? (I don't think so, but still would like a confirmation)

In the above scenario, can I possibly reduce number of directories, maybe gzip some .. Will that solve the problem?

Generally under what circumstance will mkdir fail?

Thank you

share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

Depends on which file system you're using. Reiserfs (my favorite) can handle up to 2^31 files per dir (that's 2 billion), with a max of 2^32 (4 billion) files on the filesys total. It can handle up to 64000 subdirs in a dir

Ext2 (and hence also ext3) has a limit of 32000 subdirs per dir. The max number of files per dir is theoretically unlimited (actually around 130 trillion), but performance becomes terrible with above 10-15 thousand files. The max number of total files on the filesys is limited by the number of inodes you have. With a 1 gig filesystem and a 4k block/inode ratio (the default), you have around 260000 inodes, and that's also the max number of files you can have.

http://serverfault.com/questions/98235/how-many-files-in-a-directory-is-too-many-downloading-data-from-net

share|improve this answer
    
Just to clarify, how can i tell if my system is EXT2 or not? –  Jam Aug 3 '10 at 13:41
    
I believe fdisk -l will print that out. –  Luke has no name Aug 3 '10 at 14:14
1  
[root@rajat Rajat]# parted -l Model: ATA HDS728080PLA380 (scsi) Disk /dev/sda: 82.3GB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system Flags 1 1049kB 525MB 524MB primary ext4 boot 2 525MB 4820MB 4295MB primary linux-swap(v1) 3 4820MB 82.3GB 77.5GB primary ext4 –  Rajat Aug 3 '10 at 14:36
add comment

It depends on which filesystem you use. For ext3, there is a limit on 31998 subdirectories per directory. On ext3 it's also possible to run out of inodes before running out of free space if you have lots of very small files (the ratio of blocks per inode can be set a fs creation time).

More modern filesystems typically have higher limits, or are limited only by the available disk space.

That being said, that many directories or files is typically an indication that you're doing something wrong. Consider storing data in some binary scientific format such as netcdf, hdf5, or a database.

share|improve this answer
add comment

You have an ext4 file system. The max sub directories with ext4 is 64,000.

EDIT: It can technically go higher using the "dir_nlink" feature of ext4.

share|improve this answer
    
On what do you base the statement that the OP has an ext4 filesystem? –  Dennis Williamson Aug 3 '10 at 17:44
    
Because in his reply to "Luke has no name" where he pasted the result of 'fdisk -l' it says ext4. –  Jacob K Aug 3 '10 at 19:20
    
That was actually "Rajat" and not the OP –  janneb Aug 4 '10 at 8:29
    
Ha. My bad. I'll try to pay attention next time. –  Jacob K Aug 4 '10 at 23:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.