Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need to keep millions of files on an ext4 system.
I understand that having a structure with multiple subdirectories is the general accepted solution. I wonder what would be the optimal approach in terms of number of dirs/subdirs.
For example I tried a structure like 16/16/16/16 (that is, (sub)directories from 1 to 16) and I found that I am able to move 100K files to this structure in 2m50s.
When trying to move 100K files to a 8/8/8/8/8/8 structure it took 11 minutes. So the 16/16/16/16 approach seems to be better but I was wondering if anyone has some empirical experience with an even better dir/subdir distribution.

share|improve this question

2 Answers 2

1) XFS now is better choice than EXT4.

2) Even 1024 per directory entries should be quite okay, so you can appropriately reduce level of sub-dirs depth.

share|improve this answer
up vote 0 down vote accepted

After lots of testing I figured out a solution that works for me just find and I hope it may help others as well:
I created a 255/255 dir structure and then I used the ramdisk (/dev/shm) to quickly create the directories and then save files inside them. Then once every minute I move the files to HDD.
Using this approach reduced the time to save 100K files in 255/255 directories from about 5mins to some 5 seconds.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.