5

I am trying to find the bottleneck in the rebuilding of a software raid6.

## Pause rebuilding when measuring raw I/O performance
# echo 1 > /proc/sys/dev/raid/speed_limit_min
# echo 1 > /proc/sys/dev/raid/speed_limit_max
## Drop caches so that does not interfere with measuring
# sync ; echo 3 | tee /proc/sys/vm/drop_caches >/dev/null
# time parallel -j0 "dd if=/dev/{} bs=256k count=4000 | cat >/dev/null" ::: sdbd sdbc sdbf sdbm sdbl sdbk sdbe sdbj sdbh sdbg 
4000+0 records in
4000+0 records out
1048576000 bytes (1.0 GB) copied, 7.30336 s, 144 MB/s
[... similar for each disk ...]
# time parallel -j0 "dd if=/dev/{} skip=15000000 bs=256k count=4000 | cat >/dev/null" ::: sdbd sdbc sdbf sdbm sdbl sdbk sdbe sdbj sdbh sdbg 
4000+0 records in
4000+0 records out
1048576000 bytes (1.0 GB) copied, 12.7991 s, 81.9 MB/s
[... similar for each disk ...]

So we can read sequentially at 140 MB/s in the outer tracks and 82 MB/s in the inner tracks on all the drives simultaneously. Sequential write performance is similar.

This would lead me to expect a rebuild speed of 82 MB/s or more.

# echo 800000 > /proc/sys/dev/raid/speed_limit_min
# echo 800000 > /proc/sys/dev/raid/speed_limit_max
# cat /proc/mdstat
md2 : active raid6 sdbd[10](S) sdbc[9] sdbf[0] sdbm[8] sdbl[7] sdbk[6] sdbe[11] sdbj[4] sdbi[3](F) sdbh[2] sdbg[1]
      27349121408 blocks super 1.2 level 6, 128k chunk, algorithm 2 [9/8] [UUU_UUUUU]
      [=========>...........]  recovery = 47.3% (1849905884/3907017344) finish=855.9min speed=40054K/sec

But we only get 40 MB/s. And often this drops to 30 MB/s.

# iostat -dkx 1
sdbc              0.00  8023.00    0.00  329.00     0.00 33408.00   203.09     0.70    2.12   1.06  34.80
sdbd              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdbe             13.00     0.00 8334.00    0.00 33388.00     0.00     8.01     0.65    0.08   0.06  47.20
sdbf              0.00     0.00 8348.00    0.00 33388.00     0.00     8.00     0.58    0.07   0.06  48.00
sdbg             16.00     0.00 8331.00    0.00 33388.00     0.00     8.02     0.71    0.09   0.06  48.80
sdbh            961.00     0.00 8314.00    0.00 37100.00     0.00     8.92     0.93    0.11   0.07  54.80
sdbj             70.00     0.00 8276.00    0.00 33384.00     0.00     8.07     0.78    0.10   0.06  48.40
sdbk            124.00     0.00 8221.00    0.00 33380.00     0.00     8.12     0.88    0.11   0.06  47.20
sdbl             83.00     0.00 8262.00    0.00 33380.00     0.00     8.08     0.96    0.12   0.06  47.60
sdbm              0.00     0.00 8344.00    0.00 33376.00     0.00     8.00     0.56    0.07   0.06  47.60

iostat says the disks are not 100% busy (but only 40-50%). This fits with the hypothesis that the max is around 80 MB/s.

Since this is software raid the limiting factor could be CPU. top says:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                              
38520 root      20   0     0    0    0 R   64  0.0   2947:50 md2_raid6
 6117 root      20   0     0    0    0 D   53  0.0 473:25.96 md2_resync

So md2_raid6 and md2_resync are clearly busy taking up 64% and 53% of a CPU respectively, but not near 100%.

The chunk size (128k) of the RAID was chosen after measuring which chunksize gave the least CPU penalty.

If this speed is normal: What is the limiting factor? Can I measure that?

If this speed is not normal: How can I find the limiting factor? Can I change that?

2 Answers 2

1

I don't remember exactly the speeds I had when I migrated to 6 disk RAID 6 from 4 disk RAID 5, but they were similar (4TB usable array, 24h rebuild, so around 45MB/s).

You have to remember that even the speed_limit_min will give some priority to applications that try to use the array. As such, the mechanism used to detect activity may require a 50% load on the disks to detect it and still have the ability to serve the IO requests. Did you try unmounting the partition?

To check for bottlenecks you'll have to trace the kernel (for example, using Linux Tracing Toolkit lttng, or System Tap). It's not easy and will take lots of time so unless you have to rebuild the arrays on few computers, It's probably not worth it. As for changing it: I'm sure such patches to Linux kernel will be welcome :)

1
  • The disk was inactive and had no filesystem I/O during the measuring (though it was not umounted).
    – Ole Tange
    Nov 12, 2012 at 14:00
0

I would not expect a Raid6 recovery operation to be of sequential nature since it usually needs to recover checksums and data blocks from n-1 drives which are embedded between data blocks on these drives.

In addition to this I would expect a somewhat sequential operation (=not full parallel) like:

  1. read datablock1
  2. read datablock2 ...
  3. read datablockn-1
  4. read checksum1
  5. calculate datablockn
  6. write datablockn

at least 5. is a synchronisation point so duration(1..4) is at least duration(slowest(1..4)). How well it peforms is determined by the level of parallelization of any involved layer (md, driver, controller (ncq etc)).

I would never expect a rebuild rate of a raid6 anywhere near the sequential read/write times of the single disks.

For comparison: our PS6000 Equallogic arrays (16x1TB) take around 32 hours under moderate load to rebuild a failed disk.

5
  • We know that slowest(1..4) = 82 MB/s. That includes driver and controller, but not md. If there is reasonable read-ahead cache we should be able to disregard the rotational latency. We know the checksum computation does not take full CPU power. This leaves the other layers. Can I see the time spent by these layers somewhere (without having to resort to tracing the kernel)?
    – Ole Tange
    Nov 12, 2012 at 14:16
  • 1
    Ole is right. the reads on each disk should be sequential. Each disk can be read in parallel, and then the OS can put them together serially. The disk reads and writes should be the slow part, and that part is parallelized.
    – Dan Pritts
    Nov 12, 2012 at 14:42
  • The reads are not sequential as the parity/checksum information is striped across every single disk.
    – tim
    Nov 12, 2012 at 14:57
  • To rebuild chunk 1000 on drive 9 you need to read chunk 1000 of 7 of the remaining 8 drives. To rebuild chunk 1001 you need to read chunk 1001 of 7 of the remaining 8 drives. One of the drives could skip a chunk here, but each chunk is only 128k, and the read-ahead cache is several MB, so in practice all the 8 drives will be read sequentially.
    – Ole Tange
    Nov 12, 2012 at 16:52
  • Forget my last comment, I think you are right in this aspect.
    – tim
    Nov 12, 2012 at 17:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .