2

I have a root directory 'data_0'. Under this directory are about 15,000 directories ('a', 'b', 'c', ... 'aa', 'ab' ...). Under each of theses directories there are thousands of very small files (4~10kB), something between 1,000 and 2,000 files each.

All this leads to 30 million files. I need to move these from 'data_0' to a 'data_1' folder, but without the "level 2" folders (a, b, c etc), so:

/data_0/a/1.txt --> /data_1/a_1.txt
/data_0/a/2.txt --> /data_1/a_2.txt
...
/data_0/ccc/989.txt --> /data_1/ccc_989.txt
...

How far can I go with this? Performance is unimportant here. Is there a logical limit or just a performance limit?

2 Answers 2

2

If strangers on the internet are to be trusted, there is no limit to the number of files that an ext3 folder can contain. So says the ext3-users RedHat mailing list. The 2.6 kernel supposedly allows for the theoretical allowance of "billions" of files in one directory. You may want to tweak dir_index a bit to make it run smoothly if you'll be doing in searching on the files. There are also some other side effects of massive amounts of files in one directory that you might want to read through in this StackOverflow thread.

So the answer is most likely: "Yes, but..."

1
  • hm... very tks. i already have a directory with about 300k files and performance definitely sucks for listing files and stuff, but since the files have well defined names and i do not try to list or open anything but a single file each time, i do not suffer from this performance penalty.
    – chesterman
    Aug 19, 2011 at 11:39
4

There is no logical limit to the number of files in a single directory, although there are significant performance limits as individual directories get more files (my rule-of-thumb is keep it under 1000, although I've done 10000 in certain situations). There is a limit to the total number of files on a filesystem, governed by your filesystem creation parameters (total number of inodes). There is also a hard-coded limit of 32000 subdirectories per directory (of which two entries are always taken up by . and ..).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.