I need to keep millions of files on an ext4 system.
I understand that having a structure with multiple subdirectories is the general accepted solution. I wonder what would be the optimal approach in terms of number of dirs/subdirs.
For example I tried a structure like 16/16/16/16 (that is, (sub)directories from 1 to 16) and I found that I am able to move 100K files to this structure in 2m50s.
When trying to move 100K files to a 8/8/8/8/8/8 structure it took 11 minutes. So the 16/16/16/16 approach seems to be better but I was wondering if anyone has some empirical experience with an even better dir/subdir distribution.


1) XFS now is better choice than EXT4.

2) Even 1024 per directory entries should be quite okay, so you can appropriately reduce level of sub-dirs depth.


After lots of testing I figured out a solution that works for me just find and I hope it may help others as well:
I created a 255/255 dir structure and then I used the ramdisk (/dev/shm) to quickly create the directories and then save files inside them. Then once every minute I move the files to HDD.
Using this approach reduced the time to save 100K files in 255/255 directories from about 5mins to some 5 seconds.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.