Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So on my linux server which runs Ubuntu 12.04, I have a folder in there that has ~3000 folders and within those folders they contain about 3000 text files each. In total, this takes roughly 80 GB.

So whenever I ls into any of these folders, the systems starts to get horribly slow. The system is a Xeon E5-2620 w/ 64 gb of RAM and the hard drive is on is a basic 7200 RPM Seagate HDD (don't remember if it's SATA 3.0 or not).

I'm also currently deleting the files, because they need to be regenerated. The rm process is taking fairly slow, however I'm just running a multithreaded rm -rf so there could probably be some improvement there.

Any tips you guys could give to speed this all up a bit? Thanks

share|improve this question
1  
More and better disks? –  Michael Hampton Jan 30 '13 at 0:42
    
Is dir_index enabled? sudo tune2fs -l /dev/sda1 | grep dir_index –  Jeff Ferland Jan 30 '13 at 2:23
add comment

1 Answer 1

up vote 2 down vote accepted

It is expected to be slow. See with a large number of files when you run a ls, all the inode of the files need to fetched and brought into memory. Also all the files will traverse one path long in their dentry list. Kernel will cache this information also. So, when you run a first time ls, it is bound to be slow.

A pretty simple diagnostic measure is to run ls on a new directory from one tab and then go to another tab and run slabtop -o. This will show increased cache and dentry slab use of ext4. Once it is cached, subsequent ls on the same directory will be less time consuming.

The only improvement you can do is to get more powerful disks or stripe the data. So that less disk seek has to be done, giving a performance boost.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.