Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have been through the other questions/answers regarding inode usage, mounting issues and others but none of those questions seem to apply...

df -h

/dev/sdd1 931G 100G 785G 12% /media/teradisk

df -ih

/dev/sdd1 59M 12M 47M 21% /media/teradisk

Basically, I have an EXT4 formatted drive 1TB in size, and am writing arount 12 million (12201106) files into one directory. I can't find any documentation on a files-per-directory limit for EXT4 but the filesystem reports no space left.

Oddly, I can still create new files on the drive and target folder but when doing a large cp/rsync, the calls to mkstemp and rename report no space left on device.

rsync: mkstemp "/media/teradisk/files/f.xml.No79k5" failed: No space left on device (28)

rsync: rename "/media/teradisk/files/f.xml.No79k5" -> "files/f.xml": No space left on device (28)

I know storing this many files in one directory isn't advised for a ton of reasons, but unless I can help it I don't wish to split them up.

Inode and space usage for tmpfs, the device and everything else looks fine. Any ideas of the cause?

share|improve this question
1  
Why would you need 12M files in one directory? –  rvs Apr 30 '12 at 14:28
2  
@rvs, sometimes you do not chose. –  poige Apr 30 '12 at 14:44
    
@rvs - I am limited in what I am doing. I could in theory split the files, but I am not doing directory listing or indexing of files. If I can keep them in one dir it will make my life simpler. –  Aiden Bell Apr 30 '12 at 15:40
    
So, what you've come up with? ) –  poige May 14 '12 at 4:43
    
Used XFS, solved issue. –  Aiden Bell Jun 15 '12 at 16:56

5 Answers 5

up vote 1 down vote accepted

The XFS filesystem would be a more supportable (long-term) solution for what you're trying to do now. Large file-count directories are not a problem for XFS. Of course, fixing this at the application level would also be helpful...

share|improve this answer
    
serverfault.com/questions/384541/… –  poige Apr 30 '12 at 14:30
2  
While I realize both other Filesystems cope better, and application level improvements can be made, this answer does not address the cause of the issue in ext4 other than saying "use another FS" ... Stating where ext4 and XFS differ in terms of my specific issue would have been more helpful. –  Aiden Bell Apr 30 '12 at 15:28
1  
While I did downvote this question (and my downvote is still valid) I did end up going with XFS. –  Aiden Bell Jun 15 '12 at 16:58
1  
@AidenBell Of course you did. –  ewwhite Mar 6 '13 at 23:48
    
In my embedded application, XFS, sadly, is not an option for me. Problem not solved. :'( –  Doc Sep 28 at 4:50

It seems you are hitting directory size limit. Directory itself is some kind of special file which contains names (+ inode numbers and probably some other metadata) of all files in it. And it can't be larger than 2G.

Anyway, it's not a good idea to have more than few thousands of files in one dir: searches by file name would be very slow and you'll have a lot of problems with standard tools like ls, rm and others.

Update:

a-ha!

http://old.nabble.com/re:The-maximum-number-of-files-under-a-folder-td16033098.html

On Mar 13, 2008 13:23 -0400, Theodore Ts'o wrote:

There is no limit to the number of files in a folder, except for the fact that the directory itself can't be bigger than 2GB, and the number of inodes that the entire filesystem has available to it. Of course, if you don't have directory indexing turned on, you may not like the performance of doing directory lookups, but that's a different story.

There is also a limit in the current ext3 htree code to be only 2 levels deep. Along with the 2GB limit you hit problems around 15M files, depending on the length of the filenames.

share|improve this answer
    
+1 for trying to identify the cause. Your link mentions EXT3, trying to find out if these limits remain unchanged in ext4. –  Aiden Bell Apr 30 '12 at 15:37
    
@AidenBell, I guess Reiser3 doesn't have anything like that anyway. ;) –  poige Apr 30 '12 at 16:00
3  
@poige , is ReiserFS team starting a cult? Your advocacy is ... strong. –  Aiden Bell Apr 30 '12 at 16:09
    
@AidenBell, my experience is at first. –  poige Apr 30 '12 at 16:12
    
Switched to XFS, worked without issue storing over 15m files in one directory while migrating datasets etc. –  Aiden Bell Jun 15 '12 at 16:57

is ext4 absolutely needed for you? These days XFS should handle a situation like this without a hitch.

share|improve this answer
    
I wouldn't say that. At least you need "delaylog" support and since it's quite a recent change it can be missing in OP's case. –  poige Apr 30 '12 at 14:27
1  
Have you actually tried XFS recently? With decent hardware OPs problem should be a no-problem. –  Janne Pikkarainen Apr 30 '12 at 14:32
    
This is not OP question was about. ext4 is pretty good choice and I don't think XFS will really help to deal with millions of files in one directory. –  rvs Apr 30 '12 at 14:33
    
@JannePikkarainen, I actually did (and not once). That's why I wrote a note saying that it's actually faster to copy content of XFS than remove it ( poige.livejournal.com/471842.html ) –  poige Apr 30 '12 at 14:39
1  
Again, my question isn't "Ext4 or XFS for millions of files in one directory" but "I have issue X on ext4, what could be the direct cause."; "You are using EXT4" isn't an answer to "what is the cause?" –  Aiden Bell Apr 30 '12 at 15:29

I had this problem. My solution was:

mkfs.ext4 -i 1024 -b 1024 /dev/blah

share|improve this answer
    
This would require him to reformat the partition, no? Not sure if this really helps him, he may not be able to do this. –  slm Feb 20 '13 at 14:17
    
Ext4 creates 256 byte i-nodes by default. Since your i-nodes are only keeping track of 1024 bytes, per the -i 1024 argument, 20% of your disk will be dedicated to i-nodes. If your filesystem is full of files that take the minimum 1 block of storage, this makes sense. This is extremely uncommon however, and most people will not want to use these extreme settings. –  Chris S Feb 20 '13 at 14:34
    
yup, I was testing out ext4 with the gentoo portage directory. To reproduce this error just try to rsync the gentoo portage directory into a loopback file created with the default mkfs.ext4 options. –  danbrough Feb 20 '13 at 14:46

It seems you're running out of i-nodes. Show output of df -iht ext4.

I also have had a problem with removing a directory on EXT4 containing ~ 1 million of files (linux kernel 3.0, IIRC). What's kernel version in your case?

Finally, I'd suggest using Reiser3 — it doesn't have format time i-node limits, and in aforementioned case seemed to have solved the problem as well.

UPD.: For those wondering if Reiser3 being supported on not:

cd linux-stable/fs/reiserfs && git log --pretty='format:%aD %s' . | head -n20

Tue, 10 Jan 2012 15:110:11 -0800 reiserfs: don't lock root inode searching
Tue, 10 Jan 2012 15:11:09 -0800 reiserfs: don't lock journal_init()
Tue, 10 Jan 2012 15:11:07 -0800 reiserfs: delay reiserfs lock until journal initialization
Tue, 10 Jan 2012 15:11:05 -0800 reiserfs: delete comments referring to the BKL
Mon, 9 Jan 2012 12:51:21 -0800 Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs
Wed, 21 Dec 2011 21:18:43 +0100 reiserfs: Force inode evictions before umount to avoid crash
Wed, 21 Dec 2011 17:35:34 +0100 reiserfs: Fix quota mount option parsing
Wed, 21 Dec 2011 20:17:10 +0100 reiserfs: Properly display mount options in /proc/mounts
Wed, 7 Dec 2011 18:16:57 -0500 vfs: prefer ->dentry->d_sb to ->mnt->mnt_sb
Tue, 26 Jul 2011 02:50:53 -0400 reiserfs: propagate umode_t
Tue, 26 Jul 2011 01:52:52 -0400 switch ->mknod() to umode_t
Tue, 26 Jul 2011 01:42:34 -0400 switch ->create() to umode_t
Tue, 26 Jul 2011 01:41:39 -0400 switch vfs_mkdir() and ->mkdir() to umode_t
Mon, 12 Dec 2011 15:51:45 -0500 vfs: fix the stupidity with i_dentry in inode destructors
Fri, 9 Dec 2011 08:06:57 -0500 vfs: mnt_drop_write_file()
Wed, 23 Nov 2011 11:57:51 -0500 switch a bunch of places to mnt_want_write_file()
Fri, 28 Oct 2011 14:13:29 +0200 filesystems: add set_nlink()
Fri, 28 Oct 2011 14:13:28 +0200 filesystems: add missing nlink wrappers
Tue, 25 Oct 2011 12:11:02 +0200 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
Thu, 15 Sep 2011 15:08:05 +0200 Merge branch 'master' into for-next
share|improve this answer
3  
Has the Reiser community sufficiently recovered from the whole "lead dude murdered his wife" thing to really be considered a production alternative to EXT4? –  MDMarra Apr 30 '12 at 14:19
    
@MDMarra, don't be silly. I'm talking bout Reiser3, not Reiser4. –  poige Apr 30 '12 at 14:25
4  
@poige Reiser4 is pretty advanced, having learnt lessons from previous incarnations. Version 4 now burns down the house and dumps the car in the lake after homocides. –  Dan Apr 30 '12 at 14:27
    
@Dan, have you tried? I at least did. It lacks O_DIRECT, no resizes, and alas not too stable. Hence, the only good choice of Reiser FS is version 3. –  poige Apr 30 '12 at 14:29
1  
@poige I was joking, but on a serious note, the company that is meant to develop Reiser is formally suspended with little hope of returning. It doesn't seem like the "way forward" to now begin using it in production. –  Dan Apr 30 '12 at 14:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.