I had an original software RAID 5 setup under Ubuntu 10.04.03, 3 devices, 64k chunk. Performance was acceptable:

# 4GB write
dd if=/dev/zero of=ddfile.big bs=1MB count=4k
4095290816 bytes (4.1 GB) copied, 51.743 s, 79.1 MB/s

# 4GB read
dd if=ddfile.big of=/dev/null
4095290816 bytes (4.1 GB) copied, 62.5932 s, 65.4 MB/s

Then I went to a software RAID 6 with 3 devices (2 were marked as missing, need to copy the data off them), 128k chunk. I have also set up LVM on the new array. Performance was shockingly poor:

# 4GB write
dd if=/dev/zero of=ddfile.big bs=1MB count=4k
4096000000 bytes (4.1 GB) copied, 106.406 s, 38.5 MB/s 

# 4GB read
dd if=ddfile.big of=/dev/null
4096000000 bytes (4.1 GB) copied, 129.317 s, 31.7 MB/s

I'm trying to understand what slowed those reads and writes by half. Could it be the LVM? Or the degraded state of the new raid array? Or could it be the chunk size difference (64k vs 128k)? Other than that, I'm using ext4 for the filesystem, no options.

  • 1
    These numbers are totally meaningless with the array degraded... – Chris S Aug 17 '11 at 13:24
  • 3
    LOL - "Then I went to a software RAID 6 with 3 devices (2 were marked as missing..." then "Performance was shockingly poor" - That's outstanding – Matt Simmons Aug 17 '11 at 13:35

The number of things wrong with this situation is what is shocking here.

I went to a software RAID 6 with 3 devices

A RAID-6 array has two parities. That means for every bit written there are (essentially) two computations made, one for each parity. Using RAID-6 on an array with less than 5 isn't recommended (and on less than 4 is counterproductive!). Essentially you're doing way too much work for no gain whatsoever.

2 were marked as missing

And this is where it REALLY starts to go downhill. You're doing RAID-6 with two devices missing. So you're calculating parity and putting it..nowhere? (Hey anyone with software RAID-6 experience, is this true? Does it still compute? If so, does it get dropped, or does it try to write parity to the drive?)

And any gain you would have gotten from multiple spindles is utterly eliminated because you don't have any other spindles in use.

Performance was shockingly poor

It's not really so shocking as much as self obvious at this point. You're doing tons of stuff for absolutely no benefit.

So what should you do?

Ask yourself some questions. Why do you want RAID-6? Is it because it's more fault tolerant than RAID-5? If so, you should understand which failure modes RAID-6 protects you against:

It has two parities. This comes into play whenever you suffer a drive failure, then replace the drive, then have another drive fail during rebuild. This commonly happens when you get a bad batch of drives, or when you've had drives continually exposed to high heat (probably higher than you think). In addition, there's the possibility of a Unrecoverable Read Error (URE) about once every 10^14 bits read (that's a bit over 11TB of data). If it happens during a RAID rebuild, well, that's not good, and RAID-6 can protect that.

Tell us more about the array you want to build. Why are you building it, what does the capacity need to be, and what kind of performance do you need from it? With that information, we can give you recommendations.

  • Shoot, enter just submitted the post. Here comes the rest: I have 2 x 2TB WD drives which I've been using for 6 & 12 months, they are both 99% full. I just got 3 x 2TB Seagates which I've put under software RAID 5 initially. Because the drives are somewhat similar, I wanted extra assurance hence my decision for RAID 6. I only have 3 drives available, so rather than building a RAID 5, then growing & reshaping to RAID 6, I thought that starting with a degraded RAID 6 will take less time and be less risky. – gerhard Aug 17 '11 at 16:14
  • Forgot to mention, this is a home fileserver. Lots and lots of media (pictures, music, video etc.). I'm also using it for TimeMachine back-ups (hence LVM on top of the RAID). Hardware is pretty low spec, AMD Athlon 64 X2 6000+, 3GB DDR2, old ASUS mobo, case is a CoolerMaster with lots of quiet fans for ~30C hddtemp across all drives. – gerhard Aug 17 '11 at 16:23
  • Alright, with the understanding that you're moving to 4 drives and you've only got 1 available at the moment, it's not a /good/ idea, but it'll still work (and might be the only way, short of using an external data store). RAID-10 w/ 4 drives would be better, though, in terms of performance, although there's a 1:3 chance that a drive failure during a rebuild would take out the array. It's a trade off with 4 drives, you know? – Matt Simmons Aug 17 '11 at 17:11
  • Write performance of raid 6 with 3 disks would be better than say 4 disks. Generally as you add disks read performance increases and write performance decreases. Each write requires N reads and 2 writes (where N is the number of disks). This is why raid6 and raid5 generally have very poor write performance. Now having a raid6 with 3 disks is pointless due to the wasted disk space obviously. – Yavor Shahpasov Aug 17 '11 at 18:36
  • The benefits of raid 6 are that you do not have any degradation when 1 disk is missing (faulty) degradation starts when 2 disks are missing. It also has better availability than raid5 – Yavor Shahpasov Aug 17 '11 at 18:37

Your problem is the missing disks. RAID 5 performance degrades a lot when one of the disks is missing. RAID 6 degrades when 2 of the disks are missing. Basically the system needs to regenerate the parity for each block hence the slowdown.

You should get at least as good performance after the missing disks are replaced and the raid rebuild completes.

  • OK, so the slow-down is because of the degraded RAID. Thanks for confirming this Yavor. I will get the data off one of the disks onto the new array, then add this disk to the array and see if that speeds things up. If not, I can always break up the array and re-build a RAID 5 one. Do you know if it's possible to reshape a RAID 6 into a RAID 5? – gerhard Aug 17 '11 at 16:19
  • 1
    My initial though was no. According to the internet it is possible and there are some links discussing it google "convert raid5 to raid6 md". I personally would never attempt it for production data that is not already backed up. – Yavor Shahpasov Aug 17 '11 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.