When doing some performance tests writing to a SSD-based 3-way RAID 1 mirror powered by mdadm, it appears we suffer a significant write penalty of about 2.2x slower than doing the same test on just one individual drive. We're reading from and writing to the same underlying physical drives in this test because that simulates our real world test case we're interested in.

Is this slowdown due to SATA III speed limitations or something else? I'm surprised RAID 1 would have a write penalty because I'd think it could write to all three drives simultaneously at the same speed it could write to one of them.

All three drives present:
dd if=/dev/md3 of=test.file bs=1048576 count=37193
...207.748 s, 188 MB/s

Just two drives present (i.e. normal two-drive RAID 1)
dd if=/dev/zero of=test.file bs=1048576 count=37193
...119.016 s, 328 MB/s

Just one drive present (no redundancy)
dd if=/dev/zero of=test.file bs=1048576 count=37193
...93.794 s, 416 MB/s
  • I/O channels have limits and since your using mdadm those channels go all the way back to the CPU. Note that total I/O for each case is 564 MB/s, 656 MB/s and 416 MB/s. So it could be what the SATAIII controller is connected to is the limit in the multiple drive case. The specific MB or HD controller and how the drives are connected would be needed to tell if this is the case. – Brian Aug 10 '15 at 23:18
  • @Brian If I said the server was a dual proc E5 Xeon, would it be safe to assume it's the SATA controller since that seems like a pretty slow speed for something else to be the bottleneck in that case? – sa289 Aug 10 '15 at 23:41

It appears all three SSDs are on the same controller and that you're reaching the maximum SATA III speed rating:

Third-generation SATA interfaces run with a native transfer rate of 6.0 Gbit/s; taking 8b/10b encoding into account, the maximum uncoded transfer rate is 4.8 Gbit/s (600 MB/s). (source)

Your test for writing to all three drives at the same time, so that 188MB/s is actually 564MB/sec, just short of the maximum speed.

If you can move one or more of the SSDs to separate controllers it may help.

This limitation you're hitting is one of the reasons why PCIe flash based storage is showing up in more and more high end systems.

| improve this answer | |
  • Okay, so there shouldn't be any write penalty for that type of RAID then, correct? – sa289 Aug 10 '15 at 23:42
  • There shouldn't be, but you are saturating that controller. – Gene Aug 10 '15 at 23:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.