3

I have encountered this extremely strange problem on two servers now, both running CentOS5, both of them ext4. One is a SSD, the other is a regular hard drive, both SATA without RAID.

The issue is the following, when I run rm -r on a directory with a large number of subdirectories (>1000) where each subdirectory has a large number of files (>1000), the disk where these directories reside will lock up intermittently.

This can be seen through top. Usually, the rm command will have CPU usage about 50-60% but suddenly, it will drop to zero for 10-15 seconds before returning to 50-60% for 3-4 seconds before dropping to zero again. During the time the rm command is at 0% cpu, even simple commands like ls on the drive in question will hang and nothing shows up on screen until rm is running at 50-60% again.

When rm is running at 0%, in top, I also get 0.0%wa.

As you can imagine, this constant hanging of the disk makes processing extremely slow. I am hesitant to blame it on a bad disk because I have now seen this behavior on two different systems.

Does anybody have any ideas?

EDIT: Also want to point out that when rm is running at 0.0% cpu, jbd2/sdc1-8 is still active on the disk in question.

3
  • 1
    Hi Can you run strace on the rm command like this strace -ffttTo /tmp/strace.out rm . Run this on both of the servers. It will show the syscall where time is being spent. Allow strace to run fully and don't interrupt it. Also, I think rm comes from procps. but please check from which package you are getting the binary and then provide the package version. Feb 24, 2013 at 16:25
  • 1
    can you post the smartctl results for both disks? Feb 24, 2013 at 16:45
  • I would also ask to check if the process (or which processes) are in uninterruptible sleep (check for D status on ps's output), although if you say that there's 0% usage in wa, that's weird. But worth checking too.
    – golan
    Mar 1, 2013 at 17:08

4 Answers 4

3

Not a solution but a workaround: You could start rm with ionice -c3. If you can reproduce this problem you may trace it with strace -tt -o rm.strace rm ... and contact the ext4 developers.

2
  • Hi Hauke, can you elaborate in more details exactly how this is done? I suspect this is ext4 bug as well.
    – user788171
    Feb 22, 2013 at 19:07
  • I assume that the problem can be seen in the strace output, e.g. by longer times between the file deletions. But I can't tell you what information the developers need, ask them. In theory this might even be a bug in rm, not related to ext4. What kind of CPU usage does top show, userspace load or system load? If you prepend your rm call with strace -tt -o rm.strace then strace creates a text file with timestamps and all kernel calls by rm. Just give it a try. Feb 22, 2013 at 21:33
1

Firstly,

On the ssd filesystem you will want to enable the disgard option. e.g.

 # mount -t ext4 -o discard /dev/ssd_dev /mnt/storage/location

You can read up on it here (RedHat SSD Tuning)

Lastly, you might want to review your block sizes as harddrives and SSDs sizes to differ. But If you don't want to reinstall the system, then I think a remount with the disgard option should do the trick.

Updated: The slow rm can be attributed to the filesystem write barrier as explained here

Cheers, Danie

2
  • Some people suggest to not user discard mount option; they suggest to use a cron to do the trimming. link Mar 1, 2013 at 10:14
  • If you update your answer with a bit of explanation why discard will help in this case (with your words) - you will probably get more upvotes on your answer.
    – Nils
    Mar 2, 2013 at 20:40
1

Deleting millions of files results in millions of transactions. That is going to quickly fill up the journal. The stalls you are seeing are caused by the journal being flushed.

Using a larger journal should allow more transactions to be batched up before flushing, so you should see fewer stalls like this.

The default journal size is normally 128 MB. You can use tune2fs -J size=512 on a cleanly unmounted fs to quadruple the journal size

8
  • Could also use find to batch of file deletions, and do rm -rf afterwards: find . -type f -print0|xargs -n50 -0 rm -f. That might split it up a bit.
    – lsd
    Mar 1, 2013 at 17:52
  • @lsd, that's just going to slow things down.
    – psusi
    Mar 1, 2013 at 20:24
  • BTW - Journal on SSD - is it a good idea at all to use a journal on an SSD? Can you add a good link about how the journal is being used in your answer?
    – Nils
    Mar 2, 2013 at 20:43
  • @Nils, yes, placing the journal on an SSD will help too, and if you make it large enough, enabling the data=journal mount option can speed things up too since synchronous writes can be completed as soon as the data hits the ssd, and then it can be migrated to the slow hd in the background.
    – psusi
    Mar 3, 2013 at 3:19
  • But a journal is something with a large number of write-requests. A SSD is not good at that - it is better suited for a large number of read-requests. So either put that journal onto reliable ha-RAM or a fast set of RAID instead?
    – Nils
    Mar 4, 2013 at 12:30
-1

I have found that when using the recursive option to remove a large number of files, it's best to write a simple bash script using a for loop to remove files individually. Something similar to:

for f in /path/to/dir/*
do
   # if file, delete it
   [ -f "$f" ] && rm "$f"
done
4
  • I'm not sure that this is a good solution.
    – mdpc
    Mar 1, 2013 at 16:52
  • Always open to better ideas - can you elaborate on why it's not a good solution?
    – tdk2fe
    Mar 1, 2013 at 18:47
  • Acually this will never work on a VERY large number of files, since the for-expansion will not fit into the limited allowed space. find /path/to/dir -type f -delete should do the same - without the expansion problem.
    – Nils
    Mar 2, 2013 at 20:38
  • I guess I haven't ran into a large enough scenario to encounter that - but thanks for the clarification!
    – tdk2fe
    Mar 4, 2013 at 13:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .