Every 20-30 seconds my HDD usage goes to 100% (iostat).
iotop is showing that [flush-8:0] is using 99% HDD during these times. In between HDD usage is 1-10%.
iostat output:
04/22/2013 08:58:44 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 1.55 1188.88 3.43 569.93 0.03 6.88 24.69 0.25 0.43 0.12 7.15 04/22/2013 08:58:46 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 632.50 1.00 753.50 0.01 5.41 14.72 0.77 1.02 0.02 1.35 04/22/2013 08:58:48 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 1001.00 4.50 26.50 0.04 4.01 267.74 0.08 1.63 1.15 3.55 04/22/2013 08:58:50 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 2.00 0.00 0.03 0.00 26.00 0.00 16.75 1.50 0.30 04/22/2013 08:58:52 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 2332.50 2.00 5370.00 0.03 30.04 11.46 113.70 20.79 0.15 79.30 04/22/2013 08:58:54 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 508.50 3.50 2102.00 0.03 10.21 9.96 143.96 63.78 0.47 99.50 04/22/2013 08:58:56 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 46.50 1.50 423.50 0.01 1.85 8.95 117.26 288.18 2.35 100.05 04/22/2013 08:58:58 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 924.50 3.00 34.00 0.02 3.76 209.30 1.04 203.03 1.73 6.40 04/22/2013 08:59:00 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 3.50 0.00 0.04 0.00 21.71 0.03 8.43 8.43 2.95 04/22/2013 08:59:02 AM Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 9.00 2662.50 9.00 1135.50 0.08 14.86 26.72 1.03 0.90 0.04 4.50
iotop:
[root@a18 ~]# iotop -o -a unable to set locale, falling back to the default locale Total DISK READ : 19.47 K/s | Total DISK WRITE : 2.00 M/s Actual DISK READ: 19.47 K/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 2055 be/4 root 0.00 B 1968.00 K 0.00 % 15.75 % [flush-8:0] 20991 be/4 lighttpd 7.90 M 0.00 B 0.00 % 7.05 % lighttpd -f /etc/lighttpd/lighttpd.conf 23832 be/4 root 36.00 K 714.59 M 0.00 % 6.94 % php /var/www/base/bg-worker.php
How can I figure out what is this problem caused by?
Using SSD RAID 1. Filesystem EXT4.
I have PHP server with heavy writing and lots of small files deletion.
CentOS 6 64-bit.
