Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In my organisation we are thinking about buying blade servers - instead of rack servers. Of course technology vendors also make them sound very nice. A concern, that I read very often in different forums, is, that there is a theoretical possibility of the server chassis going down - which would in consequence take all the blades down. That is due to shared infrastructure.

My reaction on this probability would be to have redundancy and by two chassis instead of one (very costly of course).

Some people (including e.g. HP Vendors) try to convince us, that the chassis is very very unlikely to fail, due to many redundancies (redundant power supply, etc.).

Another concern on my side is, that if something goes down, spare parts might be required - which is difficult in our location (Ethiopia).

So I would ask to experienced administrators, that have managed blade server: What is your experience? Do they go down as a whole - and what is the sensible shared infrastructure, that might fail?

That question could be extended to shared storage. Again I would say, that we need two storage units instead of only one - and again the vendors say, that this things are so rock solid, that no failure is expected.

Well - I can hardly believe, that such a critical infrastructure can be very reliable without redundancy - but maybe you can tell me, whether you have successfull blade-based projects, that work without redundancy in its core parts (chassis, storage...)

At the moment, we look at HP - as IBM looks much too expensive.

share|improve this question
3  
Great question. I'll post my answer and some real life failure scenarios later today. –  ewwhite Dec 2 '12 at 17:00
    
Have you looked at what Dell has in their C servers? e.g. the C6100 has 4 nodes in a 2U box, an equivalent of a 4 slot blade chassis. Instead of one 10U blade chassis you could get five 2U rack servers. No longer a single point of failure but you lose the backplane advantages. Possibly HP/IBM have an equivalent product. –  james Dec 5 '12 at 1:12

7 Answers 7

up vote 48 down vote accepted

There's a low probability of complete chassis failure...

You'll likely encounter issues in your facility before sustaining a full failure of a blade enclosure.

My experience is primarily with HP C7000 and HP C3000 blade enclosures. I've also managed Dell and Supermicro blade solutions. Vendor matters a bit. But in summary, the HP gear has been stellar, Dell has been fine, and Supermicro was lacking in quality, resiliency and was just poorly-designed. I've never experienced failures on the HP and Dell side. The Supermicro did have serious outages, forcing us to abandon the platform. On the HP's and Dells, I've never encountered a full chassis failure.

  • I've had thermal events. The air-conditioning failed at a co-location facility sending temperatures to 115°F/46°C for 10 hours.
  • Power surges and line failures: Losing one side of an A/B feed. Individual power supply failures. There are usually six power supplies in my blade setups, so there's ample warning and redundancy.
  • Individual blade server failures. One server's issues do not affect the others in the enclosure.
  • An in-chassis fire...

I've seen a variety of environments and have had the benefit of installing in ideal data center conditions, as well as some rougher locations. On the HP C7000 and C3000 side, the main thing to consider is that the chassis is entirely modular. The components are designed minimize the impact of a component failure affecting the entire unit.

Think of it like this... The main C7000 chassis is comprised of front, (passive) midplane and backplane assemblies. The structural enclosure simply holds the front and rear components together and supports the systems' weight. Nearly every part can be replaced... believe me, I've disassembled many. The main redundancies are in fan/cooling, power and networking an management. The management processors (HP's Onboard Administrator) can be paired for redundancy, however the servers can run without them.

enter image description here

Fully-populated enclosure - front view. The six power supplies at the bottom run the full depth of the chassis and connect to a modular power backplane assembly at the rear of the enclosure. Power supply modes are configurable: e.g. 3+3 or n+1. So the enclosure definitely has power redundancy. enter image description here

Fully-populated enclosure - rear view. The Virtual Connect networking modules in the rear have an internal cross-connect, so I can lose one side or the other and still maintain network connectivity to the servers. There are six hot-swappable power supplies and ten hot-swappable fans. enter image description here

Empty enclosure - front view. Note that there's really nothing to this part of the enclosure. All connections are passed-through to the modular midplane. enter image description here

Midplane assembly removed. Note the six power feeds for the midplane assembly at the bottom. enter image description here

Midplane assembly. This is where the magic happens. Note the 16 separate downplane connections: one for each of the blade servers. I've had individual server sockets/bays fail without killing the entire enclosure or affecting the other servers. enter image description here

Power supply backplane(s). 3ø unit below standard single-phase module. I changed power distribution at my data center and simply swapped the power supply backplane to deal with the new method of power delivery enter image description here

Chassis connector damage. This particular enclosure was dropped during assembly, breaking the pins off of a ribbon connector. This went unnoticed for days, resulting in the running blade chassis catching FIRE... enter image description here

Here are the charred remains of the midplane ribbon cable. This controlled some of the chassis temperature and environment monitoring. The blade servers within continued to run without incident. The affected parts were replaced at my leisure during scheduled downtime, and all was well. enter image description here

share|improve this answer
    
+1 for C7000. We've had one running for the past two years, solid, and never had any issues, hardware or performance wise, on the enclosure or the blades. –  tombull89 Dec 3 '12 at 9:04
1  
Have to agree with this - we've had a variety of blade chassis from Dell and they've been pretty much bullet-proof. I think we've had one controller module fail on one chassis and the net result of that is that we couldn't remotely manage the chassis itself for the day it took for dell support to ship us another controller and an engineer to fit it. No actual blade downtime due to the failure or the operation to replace the controller. –  RobM Dec 3 '12 at 9:27
1  
I have to agree with @ewwhite. I've been running c7000 for about 8yrs nonstop without any failures of the chassis. We even had them running in 130'F for a couple hours due to HVAC failure and had nothing fail. The big thing to keep in mind is be sure to split your power loads over multiple power panels, and to split your networking over multiple switches to eliminate a single point of failure. The only thing we've ever had go bad are some blade server hard drives, but you see this in traditional servers too. –  mrTomahawk Dec 4 '12 at 23:59

I've been managing small numbers of blade servers for eight years now, and I've yet to have a system-wide failure that took a number of blades offline. I've come real close due to power-related problems, but haven't yet had a chassis-wide failure that wasn't attributable to outside sources.

Your observation that the chassis does represent a single-point-of-failure is correct, though they do build in a large amount of redundancies in them these days. All of the blade systems I've used have had parallel power feeds to the blades, and multiple network jacks going through separate paths, and in the case of Fibre-channel multiple paths from the blade to the back-of-rack optical ports. Even the chassis information system had multiple paths.

With appropriate network engineering (redundant NIC usage, MPIO for storage) single-problem events are entirely survivable. In my time with these systems I've had the following problems, none of which affected more than one blade if any:

  • Two power-supplies fail in the blade-rack. There was enough redundancy in the other 4 to support the load.
  • Loosing a phase for a 3-phase power-supply. These supplies are rare these days, but the other two phases had enough capacity to support the load.
  • Losing an inter-chassis management loop. It was like this for years before a vendor tech on another call noticed it.
  • Losing the inter-chassis management loops entirely. We lost management-console access, but the servers kept running as if nothing was wrong.
  • Someone accidentally rebooted the back-of-rack network backplane. Everything in that chassis was using redundant NICs so experienced no interruption of service; all the traffic moved to the other backplane.

TomTom's point about cost is very true though. To get to full cost-parity your blade chassis will have to be fully loaded and likely not using special things like back-of-rack switches. Blade-racks make sense in areas where you really need the density because you're space-constrained

share|improve this answer
    
Except that SuperMicro Twin architecture gives you two computers per TU with two sockets per computer - that is similar to what you get with MOST blades. it is definitely very dense ;) The only higher density i know of are the Dell blades using ivy bridge... but they are more limited in comparison. –  TomTom Dec 2 '12 at 17:45
    
@tomtom but does the supermicro twin offer redundant psus? We just built one and I did not see that option anywhere. We bought a cold spare psu to have on hand just in case. –  Jeff Atwood Dec 2 '12 at 20:04
    
@JeffAtwood, I haven't seen redundant PSUs in SuperMicro's 1U twins, but their line of 2U 4-node twin-twins does have them. Example. –  Charles Dec 2 '12 at 20:16
    
Also, who cares. Have a spare power supply in the rack. Replacement takes seconds. –  TomTom Dec 5 '12 at 10:39

That question could be extended to shared storage. Again I would say, that we need two storage units instead of only one - and again the vendors say, that this things are so rock solid, that no failure is expected.

Actually no. You concerns so far made sense, this sentence puts them into "read the stuff in front of your eyes". HA with full replication is a known enterprise feature for storage units. Point is that a SAN (Storage unit is a lot more complex than a blade chassis that at the end is just "stupid metal". Everything in a blade chassis except some backplanes is replaceable - all modules etc. are replaceable, and individual blades ARE allowed to fail. Noone says blade center in itself gives the blades high availability.

This is a lot different from a SAN which is supposed to be up 100% of the time - in consistent state - so there you have stuff like replication etc.

THAT SAID: watch your numbers. I have considered buying blades for some time now and they NEVER MADE FINANCIAL SENSE. The chassis are just too expensive and the blades not really cheaper compared to normal computers. I would suggest looking at the SuperMicro Twin architecture as alternative.

share|improve this answer
    
Twins and twin-twins (2U 4-node) are great alternatives to blades. Intel also makes a line of twin and twin-twin servers. –  Charles Dec 2 '12 at 20:15
    
@Charles You know of th enew fat twins? 8 machines in 4 U ;) –  TomTom Dec 9 '12 at 11:13
    
I've seen one, but haven't had the chance to play with or evaluate it. –  Charles Dec 9 '12 at 15:29

The blade servers I've had experience with are those from IBM. Those particular ones are totally modular and there is much redundancy built in. So, if something is going to fail it's going to be one of the components like a PSU or a modular switch etc. But again, there is redundancy even in those.

Since being involved with the IBM blades I haven't seen a complete failure before.

With the other brands I suspect they would be constructed in a similar fashion.
It would be a good idea to speak to a vendor as well and do a lot of reading.
It is a big investment.

share|improve this answer

Failures leading to multiple blade server outages in the same enclosure are comparable (in likelihood and cause) to failures leading to multiple server outages in the same rack.

Initial set up to minimize single points of failure (two separate AC sources of power, each of which can handle the entire load, running to separate DC power supplies, such that either half can handle the entire load; two separate network attachments, either of which can handle the entire expected load, etc) and the difference between something that takes out all the blades in a chassis or all the 2U servers in a rack is very small.

share|improve this answer

A concern, that I read very often in different forums, is, that there is a theoretical possibility of the server chassis going down - which would in consequence take all the blades down. That is due to shared infrastructure.

Indeed! About 5 years ago, while managing two HP Proliant p-Class blade enclosures, I encountered chassis wide issues several times.

I've had blade servers not being able to turn on, if they had been turned off (servers are not turned off often, but became a very real problem to us just the same). I've had servers suddenly switching off and not being able to being switched on again. Finally, I had all the servers switching off and not being able to come on again.

As I recall it, pretty much all the issues were attributed to bad power backplanes or controller backplanes. We had those replaced a number of times and the non-specific, off-the-record message I got from the techs were, that they had their share of problems with this generation of blade enclosures.

I decided back then, that the benefit of blade servers were simply not worth the risk, if I had anything to say in future purchases.

Fast forward to my next employer, and my current one, for that matter. They already had HP Proliant c-Class enclosures running, so my luke warm feeling for blades didn't really matter. In the 5 years I've dealt with the c-Class enclosures, I've never experienced anything like I did with the p-Class, where an entire enclosure failed on me. They've been running without major problems.

(Except for the time a rain storm sent rain through the roof, 4 stories, a small hole in the seal of the computer room, down a cable and into the chassis)

share|improve this answer

Both DELL & HP Blade chassis lack a redundant mid-plane. This is where IBM Bladecenter proves to be a winner. To my knowledge its the only blade chassis that provides a redundant mid-plane. Though HP offers a fantastic suite of management software for the blades, we bought a Bladecenter E for our company just to avoid a single point of failure of the entire chassis.

share|improve this answer
    
This is indeed what the IBM marketing materials tell me; that they are the only vendor with a fully redundant blade solution. However after reading the other messages in this thread it seems like the HP solutions feature this as well. –  Martijn Jul 18 at 20:33

protected by Michael Hampton Oct 12 '13 at 1:25

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.