Hybrid RAID (SDD+HDD) gives unexpected results [closed]

Question

I am doing some experiments with hybrid RAID in Linux. My test consists of the following:

2x256GB SSD in RAID 0 (/dev/md1)

2x256GB HDD in RAID 0 (/dev/md2)

Then I made md1 and md2 into a RAID 1 (/dev/md127) and marking the slow HDD (md2) as --write-mostly.

Essentially, my goal is to get maximum performance AND disk space out of my SSDs, but at the same time be "safe" from drive failures. I understand that loosing one of the SSDs would mean that I fall back on slow HDDs, but that's a price I am willing to pay compared to loosing all data. Besides, it would only be for a few hours until the broken SSDs gets replaced and RAID repaired.

root@s1 / # cat /proc/mdstat
Personalities : [raid0] [raid1] [linear] [multipath] [raid6] [raid5] [raid4] [raid10]

md2 : active raid0 sdd1[1] sdc1[0]
      498802688 blocks super 1.2 512k chunks

md127 : active raid1 md1[2] md2[1](W)
      498671616 blocks super 1.2 [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

md1 : active raid0 sdb2[1] sda2[0]
      498802688 blocks super 1.2 512k chunks

Now, running a simple throghput benchmark on the 3 raid devices gives a (for me) surprising results:

root@s1 / # hdparm -t /dev/md1

/dev/md1:
 Timing buffered disk reads: 2612 MB in  3.00 seconds = 870.36 MB/sec
root@s1 / # hdparm -t /dev/md2

/dev/md2:
 Timing buffered disk reads: 812 MB in  3.01 seconds = 270.14 MB/sec
root@s1 / # hdparm -t /dev/md127

/dev/md127:
 Timing buffered disk reads: 1312 MB in  3.00 seconds = 437.33 MB/sec

RAID 0 SSD gives 870 MB/sec

RAID 0 HDD gives 270 MB/sec

RAID 1 HYBRID gives 437 MB/sec.

As the HDD raid has been marked as --write-mostly, I would assume that a pure read test would not touch the HDD at all, so what is going on here? I would assume that the hybrid benchmark would give similar results as the pure RAID 0 SSD.

At a first glance, it looks like the HDD somehow is slowing down the RAID, by being partly used for the read (even though I told it not to do reads on the HDD). However, if I have a file copy running on the HDDs while running the hdparm benchmark, I get the same result! If the HDDs WERE used, I would assume the benchmark would give even slower results if the HDDs were used for other tasks during the benchmark.

I hope some Linux raid expert can shed some light to my problem. Thanks!

Answer 1

I would assume that a pure read test would not touch the HDD at all, so what is going on here?

This is an incorrect assumption. In particular look at the read_balance() function in drivers/md/raid1.c.

 /*
   * If buffered sequential IO size exceeds optimal
   * iosize, check if there is idle disk. If yes, choose
   * the idle disk. read_balance could already choose an
   * idle disk before noticing it's a sequential IO in
   * this disk. This doesn't matter because this disk
   * will idle, next time it will be utilized after the
   * first disk has IO size exceeds optimal iosize. In
   * this way, iosize of the first disk will be optimal
   * iosize at least. iosize of the second disk might be
   * small, but not a big deal since when the second disk
   * starts IO, the first disk is likely still busy.
   */

  /*
   * If all disks are rotational, choose the closest disk. If any disk is
   * non-rotational, choose the disk with less pending request even the
   * disk is rotational, which might/might not be optimal for raids with
   * mixed ratation/non-rotational disks depending on workload.
   */

(Waves hand) If the ssd is "overloaded" then the idle hdd will be selected. /sys/block/X/queue/optimal_io_size and monitoring the number of pending requests may be helpful to determine what is happening.

If you want deep understanding you're going to have to understand read_balance().