22

We have a 16-drive RAID-6 that has three problem drives. Two are already dead, and the third is giving SMART warnings. (Nevermind how it got in such a bad state.)

Obviously we want to replace the dead drives before the one that is still working, but is it better to:

  1. replace one dead drive, let the RAID rebuild, then replace the other, and let it rebuild again; or

  2. replace both drives at once and let it rebuild both in parallel?

To put it another way, will we get back to a state of redundancy faster by reintroducing one drive or two? Does rebuilding two drives in parallel slow the rebuild process?

In case it matters, the controller is a 3ware 9650SE-16ML.

8
  • 10
    Cross everything you got that can be crossed and send your favourite $deity a large donation!
    – user9517
    Aug 30, 2011 at 17:56
  • 1
    Can I just ask one question regarding this; can you let us know the EXACT make and model of disk in this array please - if my suspicions are correct you may very well see this question become a useful referal point for future users asking certain questions. Thank you.
    – Chopper3
    Aug 30, 2011 at 17:59
  • 8
    @Warren - AIEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE! They turned it off? Bad JuJu my friend! It's too late this time, but generally speaking shutting the drives down (especially if they're older drives that have been up and running a long time) gives them an opportunity to throw errors on spinup (and cause the controller to say "Yeah, that drive is fried too now")
    – voretaq7
    Aug 30, 2011 at 18:14
  • 2
    @voretaq7: I've once sent all the disks from an MSA-20 box to hell by shutting it down after ~3 years of uptime and continuous use. Will never do that again :-)
    – karatedog
    Aug 31, 2011 at 0:53
  • 1
    The array is up and rebuilding now, so I can get exact models, for anyone who cares. The original hard drives were ST31000340NS, meaning they're the server-rated version of the one Chopper3 was asking about. So are these notorious for failing or something? (The new ones are ST31000524NS.) Sep 2, 2011 at 18:51

4 Answers 4

29

!!!!! ONE !!!!!

Do one at a time, seriously dude, don't think of doing this ANY other way ok.

Anything else will test your full system restoration skills.

8
  • 3
    The two items I would add to this answer are (1) PRAYER (to whatever deity you like) and (2) MONITORING once you get everything back to a safe state (so you'll know when drives fail in the future and can address the problem before you have two and a half failures. Optionally you can also configure a hot spare in the array for the future.
    – voretaq7
    Aug 30, 2011 at 18:22
  • 3
    Or use RAID 10 </stockanswer>
    – Chopper3
    Aug 30, 2011 at 18:24
  • 1
    Re: prayer, no comment. :) Re: monitoring, I've been advocating that for years; maybe this will light a fire under someone. Re: RAID-10, too much data in a bid market; when 3 TB disks came out, we didn't triple redundancy, we cut the number of disks by 1/3. Sigh. Re: hot spares, we do that now that drives are big enough to allow it, but this particular server was 16 drives in a 16 drive enclosure, when 1 TB drives were the biggest you could get, and we really needed all 14 available TB. Going to a 24-drive system wouldn't have worked; see previous. :) Aug 30, 2011 at 18:29
  • 2
    If the drives have already failed, then there is no reason to keep them around -- rather, I'd expect that two consecutive rebuilds are more stressful for the other drives than a single one. Aug 30, 2011 at 23:26
  • 1
    +1, This. While two consecutive rebuilds add more stress and more likely to cause the third drive to fail before you finish both, it's also a faster rebuild, and if the marginal drive fails while rebuilding the 2nd disk, you still stay online. So the fastest, safest way to a fault-tolerant state is one at a time.
    – Joel Coel
    Aug 31, 2011 at 3:21
14

Do you have good, recent backups? If not do you think you can get them in reasonable time?

I'd honestly be more concerned about tripping the bad drive offline during a rebuild than anything else - If you're already throwing SMART errors you're more than halfway there.

My suggestion would be to confirm your backups, then rebuild one drive at a time to try to recover to a state where you can replace the one throwing SMART errors (dead drives first, soft-errors last).

If you have no backups it's a crap shoot: Backing up may create enough soft errors to mark the marginal drive as failed, as may trying to do a rebuild.

3
  • 2
    Most or all of the data on this array is a kind of cache, to avoid the need to pull terabytes of data repeatedly through a slow link. This cache data is all replaceable, either by downloading it again over months (once) or by shipping it to a site that can copy from another array. So, backups aren't the issue. What we're trying to prevent by saving the array is the days to weeks of downtime shipping the server to a service depot, re-populating the array, and shipping it back. Aug 30, 2011 at 18:05
  • in that case, what @chopper3 said is pretty much The Law Of The Land: Rebuild one drive at a time and PRAY REALLY REALLY HARD that you don't trip the marginal drive offline with the extra read load.
    – voretaq7
    Aug 30, 2011 at 18:12
  • Phew - glad to hear it.
    – Chopper3
    Aug 30, 2011 at 18:19
1

I see no point in changing it as "one disk a time".

Obviously, if RAID is capable of "resilvering" both disks simultaneously (that are failed anyways) you only win allowing the whole RAID to regain its ability to sustain up to 2 failures faster.

-1

My 0.02.$

Since the server is already offline, run ddrescue on the drive that is about to fail, to clone it to another, sane drive.

Then put the new, sane drive in the array instead. If the cloning is successful, you'll avoid the risk of seeing that drive fail during 2 rebuilds.

1
  • It's a hardware RAID controller, the individual disks aren't addressable.
    – Chopper3
    Jun 26, 2016 at 8:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .