If I have 2 drives in a RAID 1, and the Raid Controller fails, does that mean the websites on the server will have downtime until the controller is replaced? Or does everything still carry on as software raid automatically until the faulty raid card is replaced?
7
-
4A fully redundant server would have multiple disks, on a dual port backplane and two controllers that can talk to a disk via either connection. This is incredibly rare in servers, but quite common in higher-end SANs where each controller can be rebooted separate to the other(s) Server redundancy is often in the physical count of servers and failover.– CriggieAug 17, 2017 at 21:08
-
RAID has a very specific purpose -- to allow you to minimize downtime due to hard drive failures.– David SchwartzAug 17, 2017 at 22:49
-
At smaller scales, MDADM and other software RAID solutions do have their advantages; namely no hardware controller to fail and replace and simplicity in moving setups from one server to another.– DamonAug 17, 2017 at 23:18
-
And at larger scales, disks are not on the machine, you'd have two SCSI cards connected to two FC switches, connected to two storage controllers.– spectrasAug 18, 2017 at 8:05
-
@Damon: Woah, we use the same name, and this... works!– DamonAug 18, 2017 at 14:20
|
Show 2 more comments
4 Answers
Or does everything still carry on as software raid automatically until the faulty raid card is replaced?
How would that work? Would the software magically bypass the hardware raid, which acts as disk controller? Not only does the OS not see the disks, they are actually physically attached to the hardware raid controller. If the controller fails, then your connection to the disks fail.
So no, you go down. Which is why small setups gain a lot from cloud offerings and large setups have multiple servers.
-
12
-
@CanadianLuke I was waiting for one of those to simply go grab a new planet, but alas...– userAug 18, 2017 at 8:46
the Raid Controller fails
I'm an electronics engineer, so my mental picture of the word "fail" may vary from a software engineers'... I mean, how often do you guys put on safety glasses for debugging?
If your RAID controller board has any kind of hardware failure which involves shorted MOSFETs, tantalum capacitors bursting into fireballs, power supply mishaps and the like, many things can happen, like your server's power supply shutting down because it detects a short. In this case everything goes down.
If the main chip on your RAID card goes dead suddenly, what happens next depends a lot on how the OS and drivers handle sudden unexpected death of a peripheral. For example, I had a PC with a dead harddisk. The PC froze for a while as the OS waited for the harddisk to respond.
People who write OS and drivers usually do this on working hardware. So the code which handles "extension card CPU just caught fire" has never been debugged. Can't blame'em. Fact of life.
There was this time when I designed a USB peripheral. I quickly found out that I had to debug that thing with a junk laptop, because if my USB peripheral misbehaved a little bit too much, the OS would just go OMFG and crash and/or reboot.
Bottom line, a card inside a PC which gets a hardware failure can have any kind of consquences, from nothing at all to kernel panic or full shutdown. So there is no way to answer your question. A RAID card failure is similar to a motherboard failure. The only thing that matters is to get your data back.
-
3That really depends on the hardware and how much effort was put in designing its failure modes. High end appliances handle hardware failures. I remember a couple of years ago a major SAN supplier making a demonstration of shooting the main storage controller with an assault rifle while streaming video and having the secondary controller pickup fast enough that video playback didn't get disrupted. Sure, you don't get that hardware in your home pc, but it's technically feasible.– spectrasAug 18, 2017 at 8:10
-
Untested code is worse than no code for handling a specific situation, no matter how hard it is to simulate. Also, lots of failure scenarios can be simulated just fine. You certainly can set up an environment where you can flip a switch to cut all traces leading to a card, effectively making the card disappear from the system. Same with chips if you want to. High end systems not uncommonly have the ability to shut off a specific expansion card port while leaving everything else running, enabling you to replace a card without needing to turn off the system. (OS support is a different matter.)– userAug 18, 2017 at 8:52
-
-
1@spectras impressive! That's hardware designed by People Who Know What They're Doing! My point was that a card inside a PC is connected to the CPU/chipset via pci-express, so for example if a MOSFET blows in the DC-DC converter on the card and it feeds +12V instead of +1.2V to the card's big fat chip, this will propagate down the pci-express lanes and also fry the cpu, chipset and whatnot. "Hardware failure" can really be anything. In your example, the boxes are probably linked via ethernet, which is transformer-isolated and, well, bulletproof, literally in that case.– bobfluxAug 18, 2017 at 20:42
-
A suggestion on how to design a raid controller that could behave just like that in SOME failure modes (eg a controller crash): Have a standby, minimal AHCI controller on board and have a FET switch assign the physical disks if it does not receive a set of watchdog pulses that tell it the main controller is still working. Would need some support in the driver to seemlessly fail over, could be helpful in remote-hosted scenarios to allow manual data salvage from a non-boot raid as is... Aug 18, 2017 at 21:47
You either setup software raid or hardware. Software can't magically pickup since the software see's two disks in a raid 1 as a single disk. It has no idea of the physical disks behind the card. It only knows what the raid card presents to the OS.
So to answer your question if the raid card fails then ya the server is going down with it. Its very rare for a raid card to fail though.
-
1Use both hardware and software RAID. Create your 2 hardware RAID arrays, then turn them into a software RAID-1 array. Should a controller fail, the other "disk" (controller) in the RAID-1 array will keep on going. Might be better off using 2 different brand controllers in case it is a bad case of manufacturing defect, etc.– ivanivanAug 17, 2017 at 19:02
-
@ivanivan You need to read the OP again - he does not have 2 hardware raid arrays to start with.– TomTomAug 17, 2017 at 19:27
If you really require that level of availability then I suggest you look into a Stratus technologies ftServer, 99.999% uptime. You can use most any OS and you don't have to change a thing in your application.
I worked on internal storage there almost 10 years ago. We can surprise remove any device (literally assert pci reset whenever) , detect the fault and failover to the secondary with zero interruption of service. It achieves this by using lockstep technology.
However, it is expensive, but not outrageous; 3-4x the cost of a equivalent server. The people who usually buy these measure their downtime in tens of thousands of dollars per second or life is at stake.
