2

We have 2 HP Lefthand SAN servers in separate data rooms. Last week each of the SANs had 1 hard disk fail. They were in different positions on the SANs. Both data rooms are very well protected from power issues with UPS.

Any ideas of what could have influenced this?

Thanks, Carl

Improve this question

6 Answers 6

Reset to default
0

Several things come to mind:

  1. your disks all share the same environment. If there was ever an event that stressed the disks, all the disks in that SAN were subjected to it. Was the shelf handled roughly when it was assembled, delivered, installed? Was there ever an overtemp event in the datacenter?
  2. Are these disks of the same manufacturing lot? Perhaps they were made when someone had a bad case of the mondays?
  3. When one drive fails, the rest of the drives in that array get stressed because the controller reads / writes like crazy to rebuild the parity. If there were other drives that were already marginal, this sudden change in utilization patterns may push them over the edge as well. As drives get larger, rebuild times get longer, and the problem gets worse.
Improve this answer
3
  • 1
    Except for item 2, these don't apply to this question, since the guy says these two disks are from two different SANs in two different rooms.
    – mfinni
    Jun 14, 2010 at 16:59
  • Looks like I had a case of the mondays...
    – chris
    Jun 14, 2010 at 18:56
  • You might want to edit your response a little given what @mfinni mentioned, and you acknowledged ;#) And also cause your answer was chosen as best by OP, so it shows up first.
    – Ellie Kesselman
    Sep 9, 2011 at 8:48
1

Sounds odd, but were they hot spare disks?

I've seen sequential (albeit not same-day scenario) failures in situations where a hot spare is spun up and has to take over. If that spare has been idle for a while, putting it to use may cause it's already-existing problems to start showing up. That's my theory at least, and I'm sticking to it! =)

Improve this answer
1
  • No, they are not hot spares, they are just part of a 12 x 500gb RAID5 SAN.
    – Carl
    Jun 14, 2010 at 15:28
1

Sounds pretty random to me, we have people who we pay to change disks, it doesn't matter what make/model/type/speed/configuration they are disks just don't work in enterprise environments any where like as long as their manufacturers make out. Keep an eye on them though.

Improve this answer
7
  • 2
    MTBF is an average, so if you see lots and lots and lots of disks, you'll see lots of failures. Plus, I suspect the disks in the MTBF tests aren't shipped around the world 3 times via USPS and UPS...
    – chris
    Jun 14, 2010 at 14:02
  • haha, no, good point, I've been doing this for over twenty years and I'm still staggered by the failure rates on fast expensive disks - I have an array for my 'play lab' with 168 x 300GB 15k disks and we seriously have to replace one a week - that's not on really is it.
    – Chopper3
    Jun 14, 2010 at 14:22
  • 1
    Yeah, we have had 5 or 6 drives fail from our 2 SANs (12 x500gb) over the past year. Is this to be expected or does it sound like a high number?
    – Carl
    Jun 14, 2010 at 15:13
  • 1
    Both failure rates (chopper3's and carl's) seem excessive given, for example, seagate's "0.63% annualized failure rate" for a 15k 300gb disk. The question is, what is a failure? Excessive SMART failures but it still stores and retrieves data? Or a loud screeching noise and smoke? And in that context, not all "failures" are the same. I'm 100% sure the 0.63% annualized failure rate is not for a single recoverable read/write error, for instance...
    – chris
    Jun 14, 2010 at 15:38
  • 1
    I'm not sure what rates a "failure" in the MTBF calculations, but I wouldn't be surprised if SAN controllers mark a disk as failed if it has too many SMART error recoveries or take a little too long to write data. I've also "repaired" many failed disks by telling the controller to "try again". Not a great solution, but if it's the only game in town (netapp filer without support in a company that had to pay me in parts and chickens because they were on their way out of business)... I've also been able to mount "failed" drives on a PC to run dban on them as well.
    – chris
    Jun 14, 2010 at 18:17
1

Looks like a case of bad luck to me so far. 24 disks, two fail in the same week? Unfortunate, but it can happen. Especially if the disks in question are 7.2K RPM drives not intended for 24x7 operation. I don't see any 500GB drives on the QuickSpecs sheets for the current LeftHand products, but I do know HP did sell 500GB 7.2K SATA drives as I have a bunch of those in an MSA1500.

If those are indeed the same drives I have, I'm not at all surprised. Those came from an earlier model of SATA drives that weren't as reliable as their SAS/SCSI counterparts, so have shown a higher failure rate here. After the first year, though, the bad apples worked their way out and I haven't had any failures since. But I was going through one a month there for a while. 5.5TB LUNs are the largest I'd suggest running with those drives, as the rebuild time (as you're probably learning right now) takes a VERY long time.

If they're really 450GB 15K SAS drives, that's much less probable but still within the realm of possibility. Sometimes these things happen.

More broadly, I know HP likes to sell LeftHand nodes in pairs. Presuming both of these units were obtained at the same time, the likelihood that all the drives are from similar batches is pretty high. As Chris S pointed out, bad batches happen. Since 5.5TB RAID5 sets can take a week to rebuild (don't have a LeftHand SAN to play with, but I know for MSA-based arrays it takes a week), and during that time the drives are under a much higher load than normal, it can cause failure cascades. However, you said "last week" which suggests they've been rebuilding for a while now and are just looking for the suspicious alignment of stars that caused the failures in the first place. If they've survived the rebuild process, it's less likely you have a seriously bad batch, maybe only slightly bad. But do keep an eye on failure rates.

2 in a week is a data-point, not a trend. Unfortunately.

Improve this answer
0

Were they both purchased at the same time? Could be from the same batch of HDs and thus share similar manufacturing anomalies. If this is the case, you should replace those drives ASAP, before any others fail.

Improve this answer
0

Disk failures on storage systems are generally preemptive disk failures. The HDD bios tracks a number read or write errors over a period of time (these can happen without interruption of service) and once a threshold is crossed the storage system marks the disk as bad before it is actually unusable (to prevent data corruption). Perhaps your storage system has a scheduled disk check for preventative maintenance. This could explain the disks being flagged as bad at around the same time.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .