Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 4 down vote favorite
1

We have an old HP DL380 G3 with 6x36GB Ultra320 SCSI drives in a RAID 10 array. We monitor the drives carefully as they are old - the SMART pre-failure indications all said OK. Then 4 days later, 2 drives had failed (in fact the drives may not have physically failed but there is corrupt data on 2 drives). How can this happen? I'm not a great believer in co-incidences and chance.

A single drive failure I can believe, but multiple failures leads me to believe there's something more going on. I have heard that the SCSI backplane on the DL380 can fail - can anyone confirm this from experience? Should we replace the SCSI backplane? Should we replace all the drives when we rebuild the array?

Edit: I have looked into some stats we had recorded recently on the accumulated run hours on the drives in this array. The 2 drives that failed had 0.6 and 2.7 run-years on them. This seems to me to discount the theory that the drives simply deteriorated at exactly the same rate and therefore failed very close together. Unless the accumulated run hours data is no more trustworthy than SMART?

share|improve this question

4 Answers

up vote 7 down vote accepted

Multiple drives failing in quick succession is not as rare as people seem to think. Failures tend to follow what's called a Bathtub curve - a high initial rate as manufacuring defects get stressed to failure, dropping to a relatively low rate for the typical lifetime of the drives and then rising again as things wear out as they pass their design lifetimes. Drives are mechanical and server drives are running constantly.

When one drive fails another failure is still only slightly more likely but such failures are typically followed by increased stress often, somewhat paradoxically, caused by the RAID rebuild process which forces the drives to carry out quite a lot of intense IO.

Finally SMART does not have a good reputation for being a reliable indicator of reliability, there is some benefit but overall it's not great - there's some very good long term study results from Google on this which you can find here (Failure Trends in a Large Disk Drive Population).

The basic message is that when you are running a RAID pack for a long time you take an increasing risk that is higher than many expect (the number of reports of multiple drive failures here is testament to that). The second message is that RAID is something to use to increase availability on average, but always make sure you have an acceptable backup strategy in case you are one of those who gets unlucky.

share|improve this answer
thx, that Google paper you link to is excellent info. – saille Jan 20 '10 at 23:41
up vote 1 down vote

Have you checked your environmental monitoring records? Any power or cooling events?

share|improve this answer
up vote 0 down vote

I don't have any experience with the HP hardware so I can't speak on that. But SMART isn't particularly good at predicting drive failures.

share|improve this answer
up vote 2 down vote

The G3 is pretty old now, I think you're seeing the other side of the MTBF bell-curve.

share|improve this answer
2  
+1. if all the disks are the same age, it's very very possible they're all going to die at the same time. I remember once when I went to hang the washing on the line and EVERY SINGLE plastic peg that worked fine last time I was out there (3 days ago) snapped in my hand. They had just been out in the sun too long and cos they were all the same age they all failed at once. – Mark Henderson♦ Jan 20 '10 at 23:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.