0

On a CentOS 7 server with 4 SSDs, I created two RAID 0 arrays with mdadm. Both were formatted with ext4 and mounted on separate directories.

I benchmarked them using fio and got the following results for random writes:

randomwrites: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.2.8
Starting 1 process
Jobs: 1 (f=1)
randomwrites: (groupid=0, jobs=1): err= 0: pid=21814: Sat May 21 15:47:19 2016
  write: io=1024.0MB, bw=696266KB/s, iops=174066, runt=  1506msec
  cpu          : usr=9.04%, sys=89.37%, ctx=3803, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=0/w=262144/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: io=1024.0MB, aggrb=696265KB/s, minb=696265KB/s, maxb=696265KB/s, mint=1506msec, maxt=1506msec

Disk stats (read/write):
    dm-0: ios=0/243409, merge=0/0, ticks=0/81182, in_queue=81298, util=93.32%, aggrios=0/262144, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    md67: ios=0/262144, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/131068, aggrmerge=0/4, aggrticks=0/43552, aggrin_queue=43553, aggrutil=92.14%
  sda: ios=0/130872, merge=0/2, ticks=0/43363, in_queue=43381, util=92.02%
  sdb: ios=0/131264, merge=0/6, ticks=0/43741, in_queue=43725, util=92.14%

Then I created a RAID 1 array with both of these RAID 0 arrays and ran the tests again.

randomwrites: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.2.8
Starting 1 process
Jobs: 1 (f=1): [w(1)] [-.-% done] [0KB/473.1MB/0KB /s] [0/121K/0 iops] [eta 00m:00s]
randomwrites: (groupid=0, jobs=1): err= 0: pid=22598: Sat May 21 16:00:55 2016
  write: io=1024.0MB, bw=482770KB/s, iops=120692, runt=  2172msec
  cpu          : usr=8.66%, sys=61.40%, ctx=73489, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=0/w=262144/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: io=1024.0MB, aggrb=482769KB/s, minb=482769KB/s, maxb=482769KB/s, mint=2172msec, maxt=2172msec

Disk stats (read/write):
    dm-0: ios=0/259433, merge=0/0, ticks=0/134614, in_queue=135499, util=95.95%, aggrios=0/262144, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    md69: ios=0/262144, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/262145, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    md67: ios=0/262145, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/131075, aggrmerge=0/0, aggrticks=0/66948, aggrin_queue=66958, aggrutil=94.64%
  sda: ios=0/130878, merge=0/0, ticks=0/66753, in_queue=66744, util=94.64%
  sdb: ios=0/131273, merge=0/0, ticks=0/67143, in_queue=67172, util=94.59%
    md68: ios=0/262145, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/131075, aggrmerge=0/0, aggrticks=0/68108, aggrin_queue=68114, aggrutil=94.68%
  sdc: ios=0/130878, merge=0/0, ticks=0/67942, in_queue=67928, util=94.68%
  sdd: ios=0/131273, merge=0/0, ticks=0/68274, in_queue=68300, util=94.68%

As you can see the RAID 0 array performed at 174066 iops while the RAID 1 over the two RAID 0s only gave 120692 iops. What is the reason for drop in write performance?

The IO scheduler is set to noop for all 4 SSDs.

1
  • A mirror will force writes to be at the slowest speed on the devices, waiting for things to catch up under high load (same as with RAID 0, but you have 4 devices now, any one of which might add a slight delay). Also, you should really rethink how you're setting up the RAID. Creating two mirrors and then striping them will offer you more redundancy at zero price or performance cost (lose one drive in a stripe, you lose the whole stripe). It's also known as RAID 10.
    – user143703
    May 21, 2016 at 20:31

1 Answer 1

1

A software RAID1 needs to duplicate each data chunk, effectively transmitting it two times down the SB and SATA link. This means that sometime, due to bus congestion a significant IOPS decrease can be observed when using high performance storage drivers (in your case, SSDs).

Try to increase the I/O queue length and/or to switch to deadline scheduler, as both changes will increase the chances of I/O coalesce, decreasing bus congestion.

3
  • Try to increase the I/O queue length - by increasing the value at /sys/block/sdX/queue/nr_requests or by some other method?
    – A.Jesin
    May 22, 2016 at 15:10
  • Yes - nr_requests is the right file.
    – shodanshok
    May 22, 2016 at 15:26
  • The default value was 128. I tried setting it to 1024, 4096 and even upto 10K and there wasn't any noticeable difference. Switching to deadline showed a decrease in performance so I switched back to NOOP.
    – A.Jesin
    May 25, 2016 at 21:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .