2

Just out of curiosity, we have a RHEL server running software RAID. There's a cronjob which kicks off a raid data check every so often to check the consistency of the array. From my understanding, the data check will utilize idle IO bandwidth so as to not interfere with the server's normal operation.

As can be seen from the iostat output below, the server is virtually idle. Why is the raid data check only using 6M/sec of IO bandwidth? During the initial RAID build, I have seen these machines reach 320M/sec+. Why is the data check not utilizing 200MB/sec as indicated as the max bandwidth in the kernel log entry?

Below is the log output from when the raid data check was last started:

Apr 28 01:00:01 xxxx kernel: md: data-check of RAID array md1
Apr 28 01:00:01 xxxx kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Apr 28 01:00:01 xxxx kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Apr 28 01:00:01 xxxx kernel: md: using 128k window, over a total of 3824596992k.
Apr 28 01:00:07 xxxx kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)

Here's some output of /proc/mdstat relating to md1:

md1 : active raid10 sdb2[1] sda2[0] sdd2[3] sdg2[6] sdh2[7] sde2[4] sdc2[2] sdf2[5]
      3824596992 blocks super 1.1 512K chunks 2 near-copies [8/8] [UUUUUUUU]
      [==========>..........]  check = 53.4% (2045997824/3824596992) finish=5255.6min speed=5639K/sec
      bitmap: 10/29 pages [40KB], 65536KB chunk

Output of iostat -m:

[root@xxxx ~]# iostat -m
Linux 2.6.32-358.2.1.el6.x86_64 (xxxx)  05/01/2013  _x86_64_    (32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.92    0.07    0.83    1.26    0.00   94.92

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sde              35.25         1.26         0.38    1520624     456526
sdf              33.15         1.21         0.38    1461256     456526
sdb              35.49         1.24         0.38    1485991     460191
sda              36.80         1.28         0.38    1542868     460191
sdc              31.64         1.26         0.36    1519241     429095
sdd              30.00         1.21         0.36    1460629     429095
sdh              30.51         1.21         0.35    1460819     424707
sdg              31.90         1.26         0.35    1519924     424707
md0               2.72         0.01         0.01       8465      12014
md1             177.21         0.38         1.34     453189    1607605

2 Answers 2

-1

This has been answered many times...it can be set with:

echo 50000 > /proc/sys/dev/raid/speed_limit_min

and

echo 50000 > /proc/sys/dev/raid/speed_limit_max
5
  • These settings seem to control the initial and rebuild operations but not the check operation. In the past, I have temporarily changed the max limit to 500000k which is how "During the initial RAID build, I have seen these machines reach 320M/sec+." On the machine which produced the output above, the settings are 1000 and 200000 respectively. By this answer, I should see near 200MB/s but that is not the case. At the very least, I should see more than 6MB/s given the above iostat output.
    – dbryant4
    May 10, 2013 at 12:47
  • @Jure1873 Note the log output, which says "max bandwidth 200000". The data check is not using anywhere near this, even while the system is idle. Thus the question. May 10, 2013 at 14:19
  • Sorry, I was reading the question to quickly. It seems something is slowing it down, but iostat shows low iowait - strange.
    – Jure1873
    May 10, 2013 at 16:04
  • what about trying blockdev --setra 16384?
    – Jure1873
    May 11, 2013 at 7:19
  • 1
    I experienced the same issue with another machine that has 16 SSDs. I set the minimum speed limit to something high in an attempt to force the OS to complete the check faster. It looks like it worked. Here is what I ran on my server: echo 10000000 > /proc/sys/dev/raid/speed_limit_min. I'm not sure why the OS chooses to take its time when checking the raid array.
    – dbryant4
    Aug 5, 2013 at 13:24
2

Take a look at /etc/sysconfig/raid-check. This lets you set the ionice and nice parameters for this check. The default is low, which nices down 5 and sets down the ionice. You can adjust this in the setting i fyou don't like that and want to set it to use more resources (at the penalty of having it interfere more with normal jobs).

1
  • To be pedantic, this directs it to take a higher priority for resources. If the I/O bandwidth it needs is not otherwise in use, it will still use it regardless of its ionice parameter. Oct 6, 2013 at 23:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .