2

I have a Perc H710 configured as a RAID5 with 4 attached 3TB Seagate 7200rpm hard drives.

Two months ago, I got these spurious errors that led me to believe the my controller was bad.

I replaced the controller and everything worked fine until a few days ago when I began to experience similar errors. Drives 00 and 03 were reported as either failed, offline, or missing. Dell sent me out yet another (3rd) RAID controller, and now the perc bios says:

Drive 00 missing
Drive 03 missing

So I pulled out the drives and examined them individually with a disk utility. Indeed, drive 00 and drive 03 have bad sectors. The Linux disk utility that I used says drive 00 has 'a few bad sectors', and drive 03 has 'many bad sectors'.

Seriously? Two drives went out on the same day?

On the other hand, is it possible that one drive failed a while back, and then another failed because it is continuously spinning, trying to rebuild the first... or something along those lines?

5
  • 7
    With multi-terabyte drives, the odds are against you. Jan 24, 2014 at 21:27
  • 4
    Surprisingly high if they're from the same batch. Jan 24, 2014 at 21:44
  • 1
    RAID 5 is as dead as dubstep
    – Chopper3
    Jan 24, 2014 at 22:42
  • 1
    @Chopper3: What should I use instead?
    – jsp
    Jan 25, 2014 at 2:45
  • 1
    @jsp 1/10 or 6/60 are really the only game in town.
    – Chopper3
    Jan 25, 2014 at 8:29

4 Answers 4

7

It's not possible to say precisely what the odds of X drives going out in Y amount of time are, but it is safe to say that drive failures are not completely independent, as commonly assumed. Multiple disk failures in the same array within close temporal proximity are actually a fairly common occurrence.

Less than a month ago, we had 4 drives fail over the same weekend on one of our production servers (same RAID set), one after another. Almost as soon as we replaced one drive, another failed... we ultimately ended up replacing all 7 drives, to be safe.

One reason, as you mentioned, is that the rebuild process is disk-intensive, so there's a non-trivial chance that a disk teetering on the edge of going bad will be pushed over the edge and fail, as a result of the increased stress it's under in providing data to rebuild the new disk.

Another factor to consider is that all the members in a RAID array tend to be in the same physical environment, and subject to very similar physical stresses (heat, vibration, power fluctuations, etc.), which tends to result in a higher incidence of similar failure times than you'd see with disks in different environments.

And, if you're like most people, you probably just bought 4 identical disks from the same place, and ended up with 4 disks from the same batch, resulting in the 4 disks sharing identical manufacturing characteristics (any defects or anomalies during that manufacturing batch are likely shared across all four disks). So identical disks in an identical environment... makes sense that they might share other similar characteristics, such as when they fail.

Finally, there's the fact that disk failures are not normally distributed (as in a bell curve). They tend to have higher failure rates at the beginning of their lives (infant mortality), and after a long period of time, when they wear out and die due to the physical stresses they've been subjected to, with a relatively lower rate of failure int he middle (the bathtub curve).

So, yes, multiple drive failures in the same RAID array happen with some regularity, and is one of the reasons you always want good backups.

6

It's actually rather common, and the main reason it is frequently advised to buy hard drives from different batches in a single RAID set. Identical batches often have identical flaws or thresholds.

Also, failures don't always result from just simple old age of the drive, they can also be triggered by minimal power surges, unexpected load for a few minutes, identical sleep spindowns etc. As such, chances are of course smaller than a single drive failure, but not that percentage squared. Also, don't forget that a single disk failure means an increased load on the other 3 because they need to work together to recalculate the missing data. This could also just push another disk over the edge. And on the same subject, a rebuild after you replace the drive is a highly intensive operation touching every sector of all disks, meaning another risky time for disks.

Finally, it may not always be the disk. I once had a RAID-5 set die on me because the controller thought 3 of the 4 disks were removed simultaneously for a few minutes. It was the controller's failure of course, but it still showed up in logs as 3 disks dying within a minute after eachother.

2

Yeah, a second failure due to disk stress from a rebuild (and the raw amount of data being read for a rebuild, with relatively high read error odds on dense modern disks) is one of the reasons that RAID-5 carries some inherent risk.

Though it sounds like the RAID controller hasn't conclusively marked either disk as failed, just "missing", this might be a case where you need to make use of your backups.

2

The problem probably is that one of your disks had a bad-block for some time, but it went unnoticed, as no program has read from this sector.

Then on another disk there was a bad sector. One of them was read and a controller removed this drive or tried to rebuild it. It then needed to read a whole second disk and encountered a second bad sector on a second drive. And there goes your RAID.

That's why it is crucial to periodically test your drives for bad sectors — so that they'll not get unnoticed for extended periods of time. There's an utility — smartd from smartmontools package — that can periodically check all disks for bad blocks while they're idling. But not all controllers allow for sending SMART commands to disks — that's why I prefer software RAID.

Disks will correct (remap) bad sectors when they'll be written to again. So if you know which sector is bad (smartctl -a can tell you) and you can check which file is using this sector, you can rewrite this file from backups to make a disk good again. But don't try to read it, as failed read can force a disk from an array.

1
  • 1
    Some RAID controllers, including PERCs have what's called "patrol read" feature aka data scrubbing, which does exactly that.
    – Mxx
    Jan 24, 2014 at 22:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .