We have a web application hosted on an internal windows vm. Our company has a number of bases around the country, each with it's own server rack, and the base LANs are linked together in a big intranet.
Presently this app is hosted on a server at HQ, but it had a power cut today. The power company scheduled it a few weeks ago and notified us by slipping a piece of paper under the door. But the country is in Covid19 lockdown so the IT contractors found out about it about an hour before it began. The first I (lead developer of this application) heard about it was an hour into the outage when the UPS's had an hour of run time left. The contractor managed to fail the vm over to another server in another base (which takes backups anyway so it was relatively straightforward to trigger another snapshot before the batteries went flat, and fire up the VM in the other base).
Anyway, while I was pretty stoked they were able to transfer it so quickly (including the internet facing hostname somehow), we hadn't done it for real before and I didn't expect it to work.
The question I have is, what's the best way to have two instances of the app running on two seperate servers that are sort of in the same intranet, but have their own independent internet connections, and have the hostname failover to the alternate server in case the primary goes down? If a reverse proxy is the answer, where do we put it? Because that's a new single point failure isn't it? It must handle the possibility of either server and it's entire base going dark, like what happened to us today.
Keeping the primary/alternate databases in sync is a problem that is easy to solve, our guys can manage that.
If you all just want to scream HOST IT IN AZURE INSTEAD that's fine, I'm keen on that too. I haven't managed to convince management that's a good idea yet. Something about data ownership they reckon. Never mind that a bunch of third party systems they use with just as confidential data is hosted in the cloud.
Finally, I'm not an infrastructure guy, the brains behind the company's IT left a few months ago and the IT contractors don't have a lot to do with high availability infrastructure. I am hoping to use this event to get some real positive improvements.