0

Recently we experienced Ec2 downtime due to AWS EAST outages.

I was told that only RDS allows "multi-az" failover, and they have no plans for that feature for EBS storage.

So the question is now:

How can we have an autoscaling option to prevent that. We have two use cases:

1) Production servers. These servers are constantly being deployed to via DeployHQ, where we push code to production and DHQ ssh's the files for us. Autoscaling when we have downtime could work, assuming we could run a script to Launch an Image; then on launch, run a shell script to git pull origin/master

.. Is that the ideal approach?

2) Production Admin servers. These are content type servers, where we cant load balance these easily. We had load balancers setup on a SHARED Ebs volume, but found that means downtime.

We could perhaps setup a master/slave approach, and Rysnc one to the other, but what happens when we have downtime (or load) and we want to autoscale. The images done nightly would be outdated, so whats recommended there?

We also dont want cron scripts running on all slaves, etc.

Improve this question
4
  • Was there a recent EBS outage in us-east-1? What kind of outage did you see? Or was it only a loss of one volume? If the latter, are you scapegoating the volume at regular intervals? For "content type servers," is there a reason not to use S3?
    – Michael - sqlbot
    Mar 16, 2016 at 23:39
  • The content is rendered by PHP (image resizing on demand)
    – Nick
    Mar 17, 2016 at 18:08
  • The content is rendered by PHP (image resizing on demand, so we cant use s3. We do use a CDN to keep the content delivery off ec2), we also run scripts and crons off the 1 box, which downloads data to process, etc; our servers were down for 15-30mins about 1 week ago. AWS SUpport said there was a problem with ebs-east# where we were located. That they had no ideas for Multi-Az supoprt to prevent it. It was a perfect storm. The EBS data was corrupting which caused issues with RDS. Rds tipped over (multi az kicked in), but by then everything was backed up and we had 1000 db conn's
    – Nick
    Mar 17, 2016 at 18:13
  • not sure why im getting downvoted for this question :)
    – Nick
    Mar 17, 2016 at 18:14

1 Answer 1

Reset to default
1

You may look into imgix to replace your on-the-fly image processing; they can do resizing and plenty more, and source images from s3 or arbitrary other places.

As to your actual question, it sounds like you need a distributed file store. Amazon recently added Elastic File Storage as one of their services, but you can also run something like GlusterFS or HDFS yourself.

Improve this answer
1
  • Just saw that - its in beta for now, so restricted to a region for now. But sorely needed. Thanks for the other recommendations
    – Nick
    Mar 21, 2016 at 17:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .