Assume you are tasked with running a set of mission critical web services with a targeted up time of 99.5%
To achieve this, you have two identical clusters of hardware in two entirely separate data centers with different bandwidth providers etc.
At the top of each rack, there needs to be some top of rack hardware (switching and firewall equipment).
Your software setup is such that, with human intervention, it is very easy to reroute all services from one of the data centers to the other in the event one of the clusters is knocked offline. Say this would take about 15 minutes.
The question is this: would you spend 8k extra per data center to have no single point of failure top of rack equipment (hot fail over switching and firewalls). By way of reference, each rack of equipment has about 50k in servers in it.
The redundant setup is much more complex, has more failure modes, and is guaranteed to cost both in terms of initial purchase price, and also in the time spent maintaining and troubleshooting the more complicated setup.
Also, we have been running the (redundant) setup for over 8 years and have never once (knock on wood) had a failure of the top of rack equipment. We replace all our stuff on 3 year in service cycles.
The downside of the single point of failure model is that if we loose a switch or a firewall, the entire data center would go down. Also, a human must throw a switch to route the failed services to the other data center (for various reasons, there is no reliable way to do this automatically)
I suspect that, because it uses less hardware and is simpler the single point of failure option will result in greater up time in the real world. It has been my experience that hardware failures occur much less frequently than people failures do and switches/firewalls (with no spinning disk drives and secure OS's etc.) rarely fail...
What does the server fault community think?