1

We have a web application hosted on an internal windows vm. Our company has a number of bases around the country, each with it's own server rack, and the base LANs are linked together in a big intranet.

Presently this app is hosted on a server at HQ, but it had a power cut today. The power company scheduled it a few weeks ago and notified us by slipping a piece of paper under the door. But the country is in Covid19 lockdown so the IT contractors found out about it about an hour before it began. The first I (lead developer of this application) heard about it was an hour into the outage when the UPS's had an hour of run time left. The contractor managed to fail the vm over to another server in another base (which takes backups anyway so it was relatively straightforward to trigger another snapshot before the batteries went flat, and fire up the VM in the other base).

Anyway, while I was pretty stoked they were able to transfer it so quickly (including the internet facing hostname somehow), we hadn't done it for real before and I didn't expect it to work.

The question I have is, what's the best way to have two instances of the app running on two seperate servers that are sort of in the same intranet, but have their own independent internet connections, and have the hostname failover to the alternate server in case the primary goes down? If a reverse proxy is the answer, where do we put it? Because that's a new single point failure isn't it? It must handle the possibility of either server and it's entire base going dark, like what happened to us today.

Keeping the primary/alternate databases in sync is a problem that is easy to solve, our guys can manage that.

If you all just want to scream HOST IT IN AZURE INSTEAD that's fine, I'm keen on that too. I haven't managed to convince management that's a good idea yet. Something about data ownership they reckon. Never mind that a bunch of third party systems they use with just as confidential data is hosted in the cloud.

Finally, I'm not an infrastructure guy, the brains behind the company's IT left a few months ago and the IT contractors don't have a lot to do with high availability infrastructure. I am hoping to use this event to get some real positive improvements.

1
  • Hire an IT guy who, after learning your infrastructure can give recommendations. Asking a bunch of people who know next-to-nothing about your use case is a waste of everyone's time. You might be able to use DNS and low TTLs to manage fail over.
    – davidgo
    Apr 21, 2020 at 10:05

1 Answer 1

1

You need a business continuity plan customized to your organization's needs. The total scope of this far exceeds one question, really this should inform all the organization's planning, not just IT.

Start by asking the organization how much unplanned downtime is tolerable for your application service levels. Perhaps this app in question is quite important and the goal is less than 60 minutes per year, 99.98% uptime. Use these service objectives to guide high availability design.

Review unplanned downtime events and determine root cause of each. Also brainstorm plausible threats that have yet to be a problem. These are your risks. Network outages including your service providers, power, malware infection, hardware failure, software failure, human mistakes, and so on.

Take power. One thing this near miss highlighted is importance of communication with the power company. Define more robust procedures for planned events. Perhaps an email list that includes IT contractors, electricians, and data center operations.

Further, generators make it possible to run without grid power. Provide an option to install generators, have enough batteries to cut over, maintain a fueling contract, and regularly test starting them up. Perhaps add secondary power with its own grid connection, batteries, generator, and distribution. Expensive, but perhaps downtime is also expensive.

Another fun power failure mode: electrical fires. Say there is smoke in the data center triggering emergency power off. After the fire department lets you back in, do you have a data center power up procedure? How long does it take? Was it ever tested?

Primary data center on fire is an excellent reason to be able to run the thing in another site. Your planned cutover was able to snapshot the app and get the very latest state. But how does that work with the primary offline? Are there backups replicated to the other location, and are they recent enough to use? Can the move be accomplished with the primary offline, given that might also take out management of backups, VM hosts, databases, DNS, and other components?

Data center outage is not easy to recover from. You need all the data already copied off site before it happens, and still be able to control things.

One design is to duplicate and isolate the entire infrastructure in each data center: replicated database, app servers, local load balancer, separate VM host clusters, and everything else. Cut over is done by a DNS or IP routing change. Advantage: isolated failure domains, as they don't depend on each other for much. Disadvantage: separate systems to maintain, cutover might be a high-impact process that takes a while to do.

Cloud does not fundamentally change business continuity planning. Only that you outsource the physical data center. And maybe you have some additional managed services to choose from.

I have not begun to cover the possible failure modes, let alone the high availability technologies that could help with rapid cutover. Keep planning, improving processes, and testing. Always keeping in mind the organization's needs for business continuity.

Not the answer you're looking for? Browse other questions tagged .