0

So on my linux server which runs Ubuntu 12.04, I have a folder in there that has ~3000 folders and within those folders they contain about 3000 text files each. In total, this takes roughly 80 GB.

So whenever I ls into any of these folders, the systems starts to get horribly slow. The system is a Xeon E5-2620 w/ 64 gb of RAM and the hard drive is on is a basic 7200 RPM Seagate HDD (don't remember if it's SATA 3.0 or not).

I'm also currently deleting the files, because they need to be regenerated. The rm process is taking fairly slow, however I'm just running a multithreaded rm -rf so there could probably be some improvement there.

Any tips you guys could give to speed this all up a bit? Thanks

2
  • 1
    More and better disks? Jan 30, 2013 at 0:42
  • Is dir_index enabled? sudo tune2fs -l /dev/sda1 | grep dir_index Jan 30, 2013 at 2:23

1 Answer 1

2

It is expected to be slow. See with a large number of files when you run a ls, all the inode of the files need to fetched and brought into memory. Also all the files will traverse one path long in their dentry list. Kernel will cache this information also. So, when you run a first time ls, it is bound to be slow.

A pretty simple diagnostic measure is to run ls on a new directory from one tab and then go to another tab and run slabtop -o. This will show increased cache and dentry slab use of ext4. Once it is cached, subsequent ls on the same directory will be less time consuming.

The only improvement you can do is to get more powerful disks or stripe the data. So that less disk seek has to be done, giving a performance boost.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .