We have some servers which run a web-app, all the servers run in a single datacenter and we've never had any issues. However as we start to get bigger I've had to think about what we're going to do if our datacenter goes down. It's not cost-effective for us to have servers in two different datacenters running 24/7 so my current plan is we have our main servers running like normal, but a 'hot' database server running on a 'cloud'/VPS server provider constantly keeping in sync with the main database servers but with no application servers connecting directly to it. Then when our datacenter goes down we clone the database server to give us enough capacity and spin up some new application servers minimizing downtime to a few minutes.
The issue I have is figuring out how to failover to the cloud servers. I don't want to use DNS round-robin because under normal conditions no requests should be going to our VPS', I also want to avoid using DNS failover (I.E. when our DC goes down we update the DNS to point to the new servers) because, in my experiance, qutie a few ISPs don't honor DNS TTLs and will cache records for days.
I'm not looking for an exact guide on how to do this, just some topics I should look into. I've looked at IP {any,multi,broad}cast but they don't seem relevant to what we're trying to do (And I'm not sure it's possible to have an IP point to multiple servers from multiple providers on different networks, but I might be wrong). I also don't want to put a proxy/load-balancer infront of all requests, again this would require equiptment in a seperate datacenter and probably reduce performance.