2

I need to keep millions of files on an ext4 system.
I understand that having a structure with multiple subdirectories is the general accepted solution. I wonder what would be the optimal approach in terms of number of dirs/subdirs.
For example I tried a structure like 16/16/16/16 (that is, (sub)directories from 1 to 16) and I found that I am able to move 100K files to this structure in 2m50s.
When trying to move 100K files to a 8/8/8/8/8/8 structure it took 11 minutes. So the 16/16/16/16 approach seems to be better but I was wondering if anyone has some empirical experience with an even better dir/subdir distribution.

2 Answers 2

2

1) XFS now is better choice than EXT4.

2) Even 1024 per directory entries should be quite okay, so you can appropriately reduce level of sub-dirs depth.

0

After lots of testing I figured out a solution that works for me just find and I hope it may help others as well:
I created a 255/255 dir structure and then I used the ramdisk (/dev/shm) to quickly create the directories and then save files inside them. Then once every minute I move the files to HDD.
Using this approach reduced the time to save 100K files in 255/255 directories from about 5mins to some 5 seconds.

1
  • What tool did you use to move files from ramdisk to HDD? mv or rsync or ? Mar 18, 2021 at 23:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .