I wanted to setup a raid01 configuration (a raid1 composed of two raid0s) with one of the raid0 volumes having the write-mostly state set so all reads would go to the other (i.e. one is disk, the other flash). However that entire plan has run into a problem in that reads directly from the raid0 are correct at 64k per disk (the chunk size), but when I add a raid1 on top of raid0, all the reads drop down to only 4k so the performance is terrible. I'm guessing that it is due to md (or something in the stack) has decided that 4k is the granularity for errors so it is doing reads in this size, but that is just a guess. Anyway I really need to find a way to fix it.
To test this I'm using a raid1 with only 1 side for simplicity i.e. that was created via
mdadm --create /dev/md2 -l 1 -n 2 /dev/md1 "missing"
Another item of interest is that the dd bs=512K on the raid0 md1 array shows 64k reads on md1 and all of its components, where I would have expected iostat to show md1 having 512K reads and its component disks to be 64K. The dd bs=512K from md2 shows 4K reads to everything. I'm computing the block size by just taking MB/s divided by tps which = MB/transaction.
Here are all the details.
[root@pe-r910 ~]# mdadm --detail /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Tue Jul 26 23:13:59 2011
Raid Level : raid1
Array Size : 1998196216 (1905.63 GiB 2046.15 GB)
Used Dev Size : 1998196216 (1905.63 GiB 2046.15 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Thu Jul 28 08:29:35 2011
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : pe-r910.ingres.prv:2 (local to host pe-r910.ingres.prv)
UUID : 299ea821:756847a0:4db591e4:38769641
Events : 160
Number Major Minor RaidDevice State
0 9 1 0 active sync /dev/md1
1 0 0 1 removed
[root@pe-r910 ~]# mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Tue Jul 26 01:05:05 2011
Raid Level : raid0
Array Size : 1998197376 (1905.63 GiB 2046.15 GB)
Raid Devices : 14
Total Devices : 14
Persistence : Superblock is persistent
Update Time : Tue Jul 26 01:05:05 2011
State : clean
Active Devices : 14
Working Devices : 14
Failed Devices : 0
Spare Devices : 0
Chunk Size : 64K
Name : pe-r910.ingres.prv:1 (local to host pe-r910.ingres.prv)
UUID : 735bd502:62ed0509:08c33e15:19ae4f6b
Events : 0
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1
4 8 81 4 active sync /dev/sdf1
5 8 97 5 active sync /dev/sdg1
6 8 113 6 active sync /dev/sdh1
7 8 129 7 active sync /dev/sdi1
8 8 145 8 active sync /dev/sdj1
9 8 161 9 active sync /dev/sdk1
10 8 177 10 active sync /dev/sdl1
11 8 193 11 active sync /dev/sdm1
12 8 209 12 active sync /dev/sdn1
13 8 225 13 active sync /dev/sdo1
[root@pe-r910 ~]# dd if=/dev/md1 bs=512K count=10000 iflag=nonblock,direct of=/dev/null
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB) copied, 3.45236 s, 1.5 GB/s
[root@pe-r910 ~]# dd if=/dev/md2 bs=512K count=10000 iflag=nonblock,direct of=/dev/null
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB) copied, 6.81182 s, 770 MB/s
[root@pe-r910 ~]#
update: This looks only to be an issue for md on md. If I make a raid1 directly on a disk, its read rate is the same as the disk. So I think I can reconfigure to be raid10 (a set of raid1's made into a radi0) instead of raid01 (2 raid0's made into a raid1).