Webserver is being blocked by some IO process

Question

My webserver is being blocked by some IO process at every 10-15 minutes when the load rises up to 50!

Usually I can see "flush-9:1" process in top using high cpu during the blocking.

This is a Debian 6.0.8 webserver with Xeon X3440, double SATA and double SSD hdd used in software raid1. The kernel is: Linux version 2.6.32-5-amd64 (Debian 2.6.32-48squeeze4) ([email protected]) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Mon Sep 23 22:14:43 UTC 2013

This server had worked fine for years, but it was not shut down properly due to an electricity outage. Since then this blocking behaviour has been appeared. I've tried to fix it with reboot, but it didn't helped.

What could be the problem?

Here is a top capture about the blocking:

top - 13:01:54 up 16 min,  2 users,  load average: 3.01, 1.43, 0.87

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31100 www-data  20   0  349m  21m 8676 S    4  0.1   0:00.02 apache2
373 root      20   0     0    0    0 D    2  0.0   0:01.79 md1_raid1
393 root      20   0     0    0    0 D    0  0.0   0:00.34 jbd2/md1-8
1267 root      20   0     0    0    0 D    0  0.0   0:07.36 flush-9:1


top - 13:02:19 up 17 min,  2 users,  load average: 4.64, 1.91, 1.04

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
1267 root      20   0     0    0    0 R   39  0.0   0:13.68 flush-9:1
1413 mysql     20   0  638m 133m 7720 S   10  0.8   6:12.61 mysqld
373 root      20   0     0    0    0 D    2  0.0   0:02.26 md1_raid1
393 root      20   0     0    0    0 D    2  0.0   0:00.82 jbd2/md1-8
31377 www-data  20   0  345m  15m 6720 S    2  0.1   0:00.01 apache2



top - 13:02:32 up 17 min,  2 users,  load average: 12.51, 3.67, 1.62
Cpu(s):  8.7%us,  7.8%sy,  0.2%ni, 78.1%id,  5.1%wa,  0.0%hi,  0.2%si,  0.0%st

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
1413 mysql     20   0  658m 139m 7720 S  425  0.9   6:33.19 mysqld
393 root      20   0     0    0    0 D   47  0.0   0:01.30 jbd2/md1-8
28326 www-data  20   0  364m  40m  14m R   35  0.3   0:00.35 apache2
31398 www-data  20   0  346m  22m  11m S   29  0.1   0:00.23 apache2
1267 root      20   0     0    0    0 R   27  0.0   0:14.27 flush-9:1
31378 www-data  20   0  345m  14m 5592 S   10  0.1   0:00.05 apache2



top - 13:02:47 up 17 min,  2 users,  load average: 13.28, 4.26, 1.84

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
1413 mysql     20   0  658m 139m 7720 S  111  0.9   6:47.93 mysqld
20347 www-data  20   0  348m  24m  13m S   24  0.2   0:00.36 apache2
31213 www-data  20   0  348m  24m  16m S   16  0.2   0:00.11 apache2
1267 root      20   0     0    0    0 D    2  0.0   0:17.29 flush-9:1
31102 www-data  20   0  347m  20m  11m S    2  0.1   0:00.06 apache2
31375 www-data  20   0  345m  20m  10m S    2  0.1   0:00.06 apache2
31437 www-data  20   0  348m  21m  11m S    2  0.1   0:00.07 apache2
373 root      20   0     0    0    0 D    0  0.0   0:02.75 md1_raid1
393 root      20   0     0    0    0 D    0  0.0   0:01.30 jbd2/md1-8


top - 13:02:50 up 17 min,  2 users,  load average: 14.38, 4.63, 1.98

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
1413 mysql     20   0  658m 139m 7720 S   34  0.9   6:49.05 mysqld
31390 www-data  20   0  348m  24m  16m S   22  0.2   0:00.19 apache2
16664 www-data  20   0  349m  28m  17m S   18  0.2   0:00.17 apache2
373 root      20   0     0    0    0 D    2  0.0   0:02.81 md1_raid1
1267 root      20   0     0    0    0 D    2  0.0   0:17.35 flush-9:1
11822 www-data  20   0  348m  32m  23m S    2  0.2   0:07.53 apache2
31306 www-data  20   0  346m  14m 6736 D    2  0.1   0:00.02 apache2
31399 www-data  20   0  347m  21m  11m S    2  0.1   0:00.07 apache2
31603 toor      20   0 19200 1412  924 R    2  0.0   0:00.01 top
393 root      20   0     0    0    0 D    0  0.0   0:01.30 jbd2/md1-8