9

I am a web developer. I have not much experience in hardware. For this reason, I use managed servers.

This morning, one of the drives in our setup failed. However, the full site went down. I asked my web host what happened and he replied that the hard disk failed in such a way that the RAID controller couldn't work properly. The array was set up as RAID 4.

Do you guys ever seen that before? Is it possible?

Thanks for any help on this guys. I need to know if my web host is honest with me.

Improve this question
3
  • If more than one disk in an Array dies then the RAID will fail (though it depends on the RAID setup).
    – Rhys Evans
    Nov 26, 2012 at 15:55
  • Short story is, your provider is an a****** and works on the cheap side. That could be perfectly acceptable as long as you as a customer have been warned that his infrastructure is not fault-tolerant do drive failures.
    – Luke404
    Nov 27, 2012 at 11:47
  • Please update the question with the raid type (i.e. raid 0,1,4,5,6,etc).
    – Trevor Boyd Smith
    Nov 27, 2012 at 13:43

7 Answers 7

Reset to default
22

More likely than not your provider is using Hard Drives that are not meant to be used in RAID. Normal consumer SATA drives fall into this category.

The likely problem is that the drive started experiencing Uncorrectable Read Errors (UREs). When this happens in a consumer drive, the drive sits there and retries the read operation (usually for 30-60 seconds) until it gives up. The RAID will wait for the drive to report the error (the 30-60) seconds. So a simple request for a few sectors can easily cause the server to grind to a halt while the failed drive grinds through those read-retry operations.

Drives that are meant for RAID Arrays have either Time Limited Error Recovery (for SATA drives). TLER reports failures back to controllers quickly, so that the controller can intelligently respond to such failures (mostly intelligently; hopefully). SCSI (SAS too) work somewhat differently. The SCSI command set allows the controller to specify various recovery effort limits on drives (MODE SELECT: RW ERR RECOVERY). A RAID controller should set the drives to fail quickly, the controller can then test if the drive thinks that it's working properly with the TUR command, fail the drive out of the array if there's a check condition.

Improve this answer
1
  • Good explanation.
    – sbrattla
    Mar 9, 2013 at 9:29
11

Yes, this is possible, even in scenarios where you would think the array should have survived the failure.

Some possibilities as to why an array fails:

  • More drives failed than could be sustained by the RAID mode. For example:
    • RAID 0 (striping) can not survive any drive failures.
    • RAID 1 can survive failures of all but 1 drive.
    • RAID 4/5 can survive 1 drive failure.
    • RAID 6 can survive 2 drive failures.
    • RAID 10 can survive the failure of up to 50% of the drives, depending on which drives fail.
  • A bug in the RAID software or controller firmware.
  • User error.
    • Someone pulled too many drives.
    • Someone pulled a drive and never replaced it, and another drive subsequently failed.
    • The array was not monitored, allowing more drives to fail than could be survived.
  • Cheap controllers with consumer grade drives are commonly known to fail even in otherwise survivable scenarios.
    • A consumer level drive will attempt almost indefinitely to read a bad sector until it gets a good read. A cheap controller will wait almost indefinitely for such a drive to return a result. The wait can be so long that the operating system gives up. Then on reboot the drives don't respond quickly enough to the controller and the array is assumed to be failed.
    • On the other hand, an enterprise level drive will give up quickly, allowing the controller to pull the data from another drive. Also, a good controller will mark a drive that takes too long to respond as failed and move on.
Improve this answer
7
  • 1
    RAID 1 should survive the death of all but one of the drives in the array. Granted, most people probably run two-drive RAID 1 setups, which means that it can only survive the death of a single drive, but that is not inherent to RAID 1.
    – user
    Nov 27, 2012 at 10:47
  • Interesting so if 1 disk in a RAID 10 fails you should break another disk because it wont survive if only one disk is broken :-) I think you should edit your post.
    – FLY
    Nov 27, 2012 at 12:50
  • @MichaelKjörling good point. I edited my post.
    – longneck
    Nov 27, 2012 at 13:44
  • @FLY you're right, i glossed over that point. edited.
    – longneck
    Nov 27, 2012 at 13:45
  • RAID4 should be RAID3. RAID3 is byte striping with parity; RAID4 was an ECC implementation needing a huge number of drives that AFAIK was never implemented.
    – Dan Is Fiddling By Firelight
    Nov 27, 2012 at 14:56
8

If it was a RAID 0 implementation then certainly when a single drive fails you'll lose the array and all of the data with it.

Improve this answer
7
  • It's a RAID 4 implementation
    – Steve Rodrigue
    Nov 26, 2012 at 15:59
  • 11
    hahaha - you nearly had me there, what is it really?
    – Chopper3
    Nov 26, 2012 at 16:04
  • 3
    @Chopper3 NetApp uses RAID4. So it's not completely unheard of, though it did give me a chuckle too. Maybe that's the host's way of saying they have a NetApp Filer or something.
    – HopelessN00b
    Nov 26, 2012 at 16:09
  • 1
    @SteveRodrigue Are you sure it's RAID 4?
    – MDMarra
    Nov 26, 2012 at 16:24
  • 1
    If it is indeed RAID4 and only 1 drive has failed, then it ought to be possible to install a new drive and rebuild the array, in principle at least. Perhaps the web host meant that one of the remaining drives failed while he was trying to do this?
    – user3490
    Nov 26, 2012 at 19:11
2

I have seen firmware bugs take out the whole RAID when a disk goes bad, or when it starts reporting imminent failure. Sorry, I have nothing specific to point you at, but yes, it can happen. Not as part of the RAID spec, of course, it is definitely a bug.

Improve this answer
1

Yes, it's possible. It's not supposed to happen, but it certainly can. Enter UREs ( Unrecoverable Read Error) and controller faults and firmware bugs and the like.

Without additional information (that your host probably won't give you), it's not possible to say definitely one way or the other, but anyone who's worked with a lot of RAID arrays has had experiences where a whole array was lost or crashed when it shouldn't have.

(And, by the way, RAID4 isn't a very commonly used RAID level, but should withstand the loss of any drive. Doesn't mean it always will, however.)

Improve this answer
1

I've had many HDD failures where not the mechanics failed, but the electronics making up the communications interface. Because of their small size many electronics components are very sensitive to even minor electrical irregularities (this can happen when large A/C motors nearby are turned on/off etc. and the power supply is a little on the cheap side).

When the drive's internal power converters or capacitors (energy storage buffers) burn out the electrical signals generated at the HDD's external connectors can and will move way out of specification. Since the drive is connected to the controller via copper wires, and often in servers many drives share a cable connection to ease installation and reduce clutter, this can easily disrupt or even permanently destroy any number of adjacent components.

This has very little to do with pricing by the way. It is true that expensive controllers and drives MAY use parts that are more tolerant to abnormal conditions or have better shielding, and that with budget components you are more likely to get sub-standard parts. But I have regularly found identical capacitors on a $50 drive and a $500 drive. And if a failed HDD directly routes 12 Volts from the power supply to the SATA connector because something shorted out, your RAID controller will be fried, no matter how many figures the price tag had.

It's not what usually happens, but it's definitely not unheard of in my experience.

Improve this answer
1
  • "often in servers many drives share a cable connection" Not in modern SAS or SATA environments. It's pretty astronomically unlikely that your scenario is what happened here; I don't think I've ever heard of a drive's electronics dying and taking other components with it. While 12v would certainly fry a SATA or SAS controller, the logic components are very rarely connected to the 12v in any way, as stepping the voltage down from 12 to 3.3 or less is very complicated compared to 5v or 3.3v sources. I'm curious where you might have head of this sort of thing happening; if you're willing to share?
    – Chris S
    Nov 27, 2012 at 2:49
1

Yes, I guess the whole raid can fail after a single drive failure. The first failing drive will be taken offline by the controller and the raid will still work fine. But when the failed drive is replaced, the controller starts rebuilding the raid. If there is a latent non-discovered read-problem on one of the other remaining drives, a rebuild of the failed drive could cause more drives to go offline (when read-problems are discovered while rebuilding the raid) again causing the whole raid to fail.

Improve this answer
1
  • This is why RAID arrays need to be scrubbed regularly, to discovery read or write problems.
    – Chris S
    Nov 27, 2012 at 15:03

Not the answer you're looking for? Browse other questions tagged .