High Availability on a Budget using EC2 for Fail-Over?

Question

I am doing some research to determine how to best create a high availability solution for deployment of a rails/mysql web application for the minimum cost.

I've come up with a possibility, and am looking for feedback on it and additional ideas or possibilities.

I want my primary production instance, to be co-located servers. This greatly increases the amount of power I can get per dollar of hosting, while still having a pretty secure and reliable hosting environment. I've found a couple of local companies that will fit the bill nicely. I am not totally great at it but I can build and rack servers, and I am a competent admin so I can live without a lot of the nicer managed services.

I do not want to have to buy a complete disaster recovery/fail-over instance with the same specs - for one thing it should be in a different region which would mean it would have to be dedicated or virtual because I can't get into the data-center; and would consequently be much more expensive than my co-located instance and would most likely not be used.

So my thought is use EC2 for fail-over, and leave the instances offline until they are needed. So fail-over would not be instantaneous but the new servers would be up in a few minutes. I would still need some instance online all the time though, to act as a MySQL replication slave target.

I am not sure of the best way to manage the fail-over however. I have a couple thoughts: one is to use TZO-HA which can just be used for port test and fail-over but of course it only does it at a DNS level which would mean there would still be an outage until people's DNS cache expires.

So my thought then is to put a low-end EC2 instance up that can run HAProxy. The default configuration would be elastic IPs to this instance which proxies to my co-located server. There are a couple different ways to automate the actual fail-over if it occurs either with the co-located application server or the HAProxy nearly instantaneously.

Disadvantage of course is that this creates an extra hop for every single request. And, I'll be putting a lot of bandwidth through EC2 eventually which begs the question of why not just use them to begin with since bandwidth is a large portion of my estimated hosting costs there.

Any problems with my plan? Any other ideas?

Answer 1

I think AWS allows you to set up their own load balancers (ELB) which will also allow you to set up automatic rules for launching more instances of your AMIs as the load demands (and shutting them down again afterwards).

I haven't implemented it myself, though I've been looking in to it. I did find this fantastic guide to setting up load-balanced LAMP stacks: http://www.lindstromconsulting.com/node/7 (The AWS layout has changed a little since this was written, but it's still incredibly useful).

(Edit: link has moved, now that I look: http://bit.ly/aaasX2)