Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a 16-drive RAID-6 that has three problem drives. Two are already dead, and the third is giving SMART warnings. (Nevermind how it got in such a bad state.)

Obviously we want to replace the dead drives before the one that is still working, but is it better to:

  1. replace one dead drive, let the RAID rebuild, then replace the other, and let it rebuild again; or

  2. replace both drives at once and let it rebuild both in parallel?

To put it another way, will we get back to a state of redundancy faster by reintroducing one drive or two? Does rebuilding two drives in parallel slow the rebuild process?

In case it matters, the controller is a 3ware 9650SE-16ML.

share|improve this question
7  
Cross everything you got that can be crossed and send your favourite $deity a large donation! –  Iain Aug 30 '11 at 17:56
1  
Can I just ask one question regarding this; can you let us know the EXACT make and model of disk in this array please - if my suspicions are correct you may very well see this question become a useful referal point for future users asking certain questions. Thank you. –  Chopper3 Aug 30 '11 at 17:59
    
I don't know the exact model, and can't query the controller to find out, because someone else decided the machine should be turned off until the replacement drives get there. From data captured before then, I can say they're 1 TB Seagates with serials beginning with 9QJ. –  Warren Young Aug 30 '11 at 18:08
8  
@Warren - AIEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE! They turned it off? Bad JuJu my friend! It's too late this time, but generally speaking shutting the drives down (especially if they're older drives that have been up and running a long time) gives them an opportunity to throw errors on spinup (and cause the controller to say "Yeah, that drive is fried too now") –  voretaq7 Aug 30 '11 at 18:14
2  
@voretaq7: I've once sent all the disks from an MSA-20 box to hell by shutting it down after ~3 years of uptime and continuous use. Will never do that again :-) –  karatedog Aug 31 '11 at 0:53

2 Answers 2

up vote 23 down vote accepted

!!!!! ONE !!!!!

Do one at a time, seriously dude, don't think of doing this ANY other way ok.

Anything else will test your full system restoration skills.

share|improve this answer
2  
The two items I would add to this answer are (1) PRAYER (to whatever deity you like) and (2) MONITORING once you get everything back to a safe state (so you'll know when drives fail in the future and can address the problem before you have two and a half failures. Optionally you can also configure a hot spare in the array for the future. –  voretaq7 Aug 30 '11 at 18:22
2  
Or use RAID 10 </stockanswer> –  Chopper3 Aug 30 '11 at 18:24
1  
Re: prayer, no comment. :) Re: monitoring, I've been advocating that for years; maybe this will light a fire under someone. Re: RAID-10, too much data in a bid market; when 3 TB disks came out, we didn't triple redundancy, we cut the number of disks by 1/3. Sigh. Re: hot spares, we do that now that drives are big enough to allow it, but this particular server was 16 drives in a 16 drive enclosure, when 1 TB drives were the biggest you could get, and we really needed all 14 available TB. Going to a 24-drive system wouldn't have worked; see previous. :) –  Warren Young Aug 30 '11 at 18:29
2  
If the drives have already failed, then there is no reason to keep them around -- rather, I'd expect that two consecutive rebuilds are more stressful for the other drives than a single one. –  Simon Richter Aug 30 '11 at 23:26
1  
+1, This. While two consecutive rebuilds add more stress and more likely to cause the third drive to fail before you finish both, it's also a faster rebuild, and if the marginal drive fails while rebuilding the 2nd disk, you still stay online. So the fastest, safest way to a fault-tolerant state is one at a time. –  Joel Coel Aug 31 '11 at 3:21

Do you have good, recent backups? If not do you think you can get them in reasonable time?

I'd honestly be more concerned about tripping the bad drive offline during a rebuild than anything else - If you're already throwing SMART errors you're more than halfway there.

My suggestion would be to confirm your backups, then rebuild one drive at a time to try to recover to a state where you can replace the one throwing SMART errors (dead drives first, soft-errors last).

If you have no backups it's a crap shoot: Backing up may create enough soft errors to mark the marginal drive as failed, as may trying to do a rebuild.

share|improve this answer
2  
Most or all of the data on this array is a kind of cache, to avoid the need to pull terabytes of data repeatedly through a slow link. This cache data is all replaceable, either by downloading it again over months (once) or by shipping it to a site that can copy from another array. So, backups aren't the issue. What we're trying to prevent by saving the array is the days to weeks of downtime shipping the server to a service depot, re-populating the array, and shipping it back. –  Warren Young Aug 30 '11 at 18:05
    
in that case, what @chopper3 said is pretty much The Law Of The Land: Rebuild one drive at a time and PRAY REALLY REALLY HARD that you don't trip the marginal drive offline with the extra read load. –  voretaq7 Aug 30 '11 at 18:12
    
I'll accept his answer, then. Thanks. –  Warren Young Aug 30 '11 at 18:15
    
Phew - glad to hear it. –  Chopper3 Aug 30 '11 at 18:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.