I am doing some research to determine how to best create a high availability solution for deployment of a rails/mysql web application for the minimum cost.
I've come up with a possibility, and am looking for feedback on it and additional ideas or possibilities.
I want my primary production instance, to be co-located servers. This greatly increases the amount of power I can get per dollar of hosting, while still having a pretty secure and reliable hosting environment. I've found a couple of local companies that will fit the bill nicely. I am not totally great at it but I can build and rack servers, and I am a competent admin so I can live without a lot of the nicer managed services.
I do not want to have to buy a complete disaster recovery/fail-over instance with the same specs - for one thing it should be in a different region which would mean it would have to be dedicated or virtual because I can't get into the data-center; and would consequently be much more expensive than my co-located instance and would most likely not be used.
So my thought is use EC2 for fail-over, and leave the instances offline until they are needed. So fail-over would not be instantaneous but the new servers would be up in a few minutes. I would still need some instance online all the time though, to act as a MySQL replication slave target.
I am not sure of the best way to manage the fail-over however. I have a couple thoughts: one is to use TZO-HA which can just be used for port test and fail-over but of course it only does it at a DNS level which would mean there would still be an outage until people's DNS cache expires.
So my thought then is to put a low-end EC2 instance up that can run HAProxy. The default configuration would be elastic IPs to this instance which proxies to my co-located server. There are a couple different ways to automate the actual fail-over if it occurs either with the co-located application server or the HAProxy nearly instantaneously.
Disadvantage of course is that this creates an extra hop for every single request. And, I'll be putting a lot of bandwidth through EC2 eventually which begs the question of why not just use them to begin with since bandwidth is a large portion of my estimated hosting costs there.
Any problems with my plan? Any other ideas?