Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

The goal is to have as much redundancy as possible so that we can quickly recover from instance failures or any other issues. Our current AWS setup for a Rails WebApp that we run is as follows:

  • EC2 instance where we have an AMI of the initial setup. We also create new AMIs every time we make a change to the instance configuration.
  • Codebase, instance user directories, and some config files live on a EBS volume (files are rsynced nightly or symlinked) which is mounted by the EC2 instance
  • Full weekly backups of EBS volume to S3 bucket and nightly incremental backups
  • WebApp uses the RDS hosted database set to 10 days of recoverability

Anyone have any suggestions for making this even more robust? (We almost got caught with our pants down and don't want it to happen again...)

share|improve this question
    
With this setup you almost got caught or before this setup you almost got caught? –  Rob Olmos Oct 31 '10 at 19:39
add comment

2 Answers

You could also run multiple EC2 instances and load balance between them. Also that should put your uptime above 99.99% if it isn't already there. From what I understand a single EC2 instance is around 99.95 or 99.5%+ of uptime, load balance them and it rockets up to 99.99% or better, often hitting the 99.999%+ range. Either way that could help also...

...and another idea is to leave an EC2 hot swappable. Also possibly keep an RDS repository hot swappable too, just in case the production copy gets corrupted or some such issue.

...and another idea, keep an EC2 instance booted with mySQL, as a last ditch emergency copy if something is up with the RDS copy.

...and last idea, run a hot swappable or load balanced - or hot swappable AND load balanced cloud hosted service in a secondary cloud such as Rackspace or other RoR friendly cloud host.

Hope that helps. :)

share|improve this answer
    
Thank you, Adron! –  Chris Nov 1 '10 at 2:05
add comment

I want to second, Adron. If you need more availability, then load balancing is the way to go.

Suggested setup

  1. Get a hosted DNS service like dnsmadeeasy or dyndns. [The idea is to get a global service that allows round-robin-loadbalancing with extremely low TTLs. An API is a plus for smooth integration.)
  2. Two elastic loadbalancers (ELB) for failover.
  3. An array of application servers behind them.

Of course the ELBs need health checks for the app servers configured.

If ELBs don't work for you (e.g. limited throughput or too eventually consistent ;-)), then look into Haproxy. It's a great piece of software. You'd have to run this on two seperate instances -- I'd go with either c1.medium or m1.small type of instances.

Monitoring & Alerting

Also kind of important is monitoring setup on all components. Not just for alerting but also for capacity planning.

I'd suggest you look into AWS CloudWatch or a service like ServerDensity. It depends how much you want to build in house, e.g. it's easier to get into ServerDensity and setup monitoring and alerting while CloudWatch requires another solution to pull the data and analyze it and then feed it into alerting.

share|improve this answer
    
Unless they changed it recently, you can only have 1 ELB pointing to an instance at any one time. However, the ELB is not a single device like a traditional load balancer, but a configuration on the boundary of Amazon's infrastructure, so the redundancy should be included already. –  Flashman Jan 4 '11 at 18:22
    
Good call. To make an ELB even more redundant you can specify that it runs in multiple availability zones: docs.amazonwebservices.com/ElasticLoadBalancing/latest/… –  Till Jan 7 '11 at 21:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.