My webserver is being blocked by some IO process at every 10-15 minutes when the load rises up to 50!
Usually I can see "flush-9:1" process in top using high cpu during the blocking.
This is a Debian 6.0.8 webserver with Xeon X3440, double SATA and double SSD hdd used in software raid1. The kernel is: Linux version 2.6.32-5-amd64 (Debian 2.6.32-48squeeze4) ([email protected]) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Mon Sep 23 22:14:43 UTC 2013
This server had worked fine for years, but it was not shut down properly due to an electricity outage. Since then this blocking behaviour has been appeared. I've tried to fix it with reboot, but it didn't helped.
What could be the problem?
Here is a top capture about the blocking:
top - 13:01:54 up 16 min, 2 users, load average: 3.01, 1.43, 0.87 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31100 www-data 20 0 349m 21m 8676 S 4 0.1 0:00.02 apache2 373 root 20 0 0 0 0 D 2 0.0 0:01.79 md1_raid1 393 root 20 0 0 0 0 D 0 0.0 0:00.34 jbd2/md1-8 1267 root 20 0 0 0 0 D 0 0.0 0:07.36 flush-9:1 top - 13:02:19 up 17 min, 2 users, load average: 4.64, 1.91, 1.04 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1267 root 20 0 0 0 0 R 39 0.0 0:13.68 flush-9:1 1413 mysql 20 0 638m 133m 7720 S 10 0.8 6:12.61 mysqld 373 root 20 0 0 0 0 D 2 0.0 0:02.26 md1_raid1 393 root 20 0 0 0 0 D 2 0.0 0:00.82 jbd2/md1-8 31377 www-data 20 0 345m 15m 6720 S 2 0.1 0:00.01 apache2 top - 13:02:32 up 17 min, 2 users, load average: 12.51, 3.67, 1.62 Cpu(s): 8.7%us, 7.8%sy, 0.2%ni, 78.1%id, 5.1%wa, 0.0%hi, 0.2%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1413 mysql 20 0 658m 139m 7720 S 425 0.9 6:33.19 mysqld 393 root 20 0 0 0 0 D 47 0.0 0:01.30 jbd2/md1-8 28326 www-data 20 0 364m 40m 14m R 35 0.3 0:00.35 apache2 31398 www-data 20 0 346m 22m 11m S 29 0.1 0:00.23 apache2 1267 root 20 0 0 0 0 R 27 0.0 0:14.27 flush-9:1 31378 www-data 20 0 345m 14m 5592 S 10 0.1 0:00.05 apache2 top - 13:02:47 up 17 min, 2 users, load average: 13.28, 4.26, 1.84 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1413 mysql 20 0 658m 139m 7720 S 111 0.9 6:47.93 mysqld 20347 www-data 20 0 348m 24m 13m S 24 0.2 0:00.36 apache2 31213 www-data 20 0 348m 24m 16m S 16 0.2 0:00.11 apache2 1267 root 20 0 0 0 0 D 2 0.0 0:17.29 flush-9:1 31102 www-data 20 0 347m 20m 11m S 2 0.1 0:00.06 apache2 31375 www-data 20 0 345m 20m 10m S 2 0.1 0:00.06 apache2 31437 www-data 20 0 348m 21m 11m S 2 0.1 0:00.07 apache2 373 root 20 0 0 0 0 D 0 0.0 0:02.75 md1_raid1 393 root 20 0 0 0 0 D 0 0.0 0:01.30 jbd2/md1-8 top - 13:02:50 up 17 min, 2 users, load average: 14.38, 4.63, 1.98 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1413 mysql 20 0 658m 139m 7720 S 34 0.9 6:49.05 mysqld 31390 www-data 20 0 348m 24m 16m S 22 0.2 0:00.19 apache2 16664 www-data 20 0 349m 28m 17m S 18 0.2 0:00.17 apache2 373 root 20 0 0 0 0 D 2 0.0 0:02.81 md1_raid1 1267 root 20 0 0 0 0 D 2 0.0 0:17.35 flush-9:1 11822 www-data 20 0 348m 32m 23m S 2 0.2 0:07.53 apache2 31306 www-data 20 0 346m 14m 6736 D 2 0.1 0:00.02 apache2 31399 www-data 20 0 347m 21m 11m S 2 0.1 0:00.07 apache2 31603 toor 20 0 19200 1412 924 R 2 0.0 0:00.01 top 393 root 20 0 0 0 0 D 0 0.0 0:01.30 jbd2/md1-8
cat /proc/mdstat