Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using Ubuntu 12.04 with 2 x 2TB SATA3 harddisks in software RAID1. The filesystem is ext4 with data=writeback.

When using find . -maxdepth 1 -name '*' -delete to delete a large number of files in a directory, doing df at intervals show that:

Initially a large number of files get deleted, and iotop shows the find operation taking most of the IO.

enter image description here

Later df shows that the number of files remain constant, and iotop shows that IO usage is now dominated by jbd2 and md4_raid1.

enter image description here

Few minutes later, df shows the number of files dropping once more.

Can anyone explain this behavior? Is this normal or is there something wrong with my server? Also, how can 3 processes all have 99.99% IO usage at the same time?

share|improve this question

2 Answers 2

It's what I would expect. Find finds files - i.e. it reads from the VFS, when the data is not in cache it needs to be fetched from disk. As it processes each file, it enqueues the changes to be written. At some point the write buffer gets full and the changes must be written to the disk - at this point the process doing the writes will be blocked making further write OR READ requests until there's space to buffer more writes.

share|improve this answer
    
When this mass deleting is done, the server slows to an almost complete halt. Is there a way for the server to avoid getting thrashed by the write block? –  Nyxynyx Mar 14 '13 at 13:43
    
Don't delete so many files all at once? –  Michael Hampton Mar 14 '13 at 13:50
    
See serverfault.com/questions/487599/… –  symcbean Mar 14 '13 at 14:25

You can prevent the system from being blocked by reducing the I/O priority of the process:

ionice -c3 nice -15 find ... -delete

If that's not enough then you can use chrt to put the process into the SCHED_IDLE class.

share|improve this answer
    
Should I actually reduce the IO priority of [jbd2/*] and [md4_raid1]? Will that slow down my entire IO subsystem even more? –  Nyxynyx Mar 15 '13 at 17:00
    
Don't touch that kernel stuff. I even doubt that it's possible to change their priority anyway. –  Hauke Laging Mar 15 '13 at 17:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.