I have encountered this extremely strange problem on two servers now, both running CentOS5, both of them ext4. One is a SSD, the other is a regular hard drive, both SATA without RAID.
The issue is the following, when I run rm -r on a directory with a large number of subdirectories (>1000) where each subdirectory has a large number of files (>1000), the disk where these directories reside will lock up intermittently.
This can be seen through top. Usually, the rm command will have CPU usage about 50-60% but suddenly, it will drop to zero for 10-15 seconds before returning to 50-60% for 3-4 seconds before dropping to zero again. During the time the rm command is at 0% cpu, even simple commands like ls on the drive in question will hang and nothing shows up on screen until rm is running at 50-60% again.
When rm is running at 0%, in top, I also get 0.0%wa.
As you can imagine, this constant hanging of the disk makes processing extremely slow. I am hesitant to blame it on a bad disk because I have now seen this behavior on two different systems.
Does anybody have any ideas?
EDIT: Also want to point out that when rm is running at 0.0% cpu, jbd2/sdc1-8 is still active on the disk in question.