4

Currently our system runs entirely inside AWS. We do rolling snapshots of our EBS and often practice running restores.

What keeps me up at night is having all our eggs in one basket. Here are the scenario's:

  • Our Amazon zone has some massive event that destroys the data centre
  • Someone gains access to our AWS account, terminates our instances and deletes all our snapshots

To mitigate these risks I'm thinking about moving snapshots periodically to another AWS account (with different credentials) in another region.

My question is this, is this an adequate level of precaution or should I be looking to offsite backups that are completely removed from Amazon?

8

This is risk assessment, not professional systems administration. There's, arguably, a technical component to this decision, but it's fundamentally a business (and dollars and cents) decision.

I think it's wise to plan for both scenarios if it can be handled cost-effectively. The second scenario seems vastly more likely than the first, but they're both plausible.

If it were me, I'd lobby hard for offsite backups that were completely removed from Amazon. A third, perhaps more likely scenario than your first two, might involve the business relationship between your company and Amazon going sour. While there are certainly legal remedies in that situation it would be advantageous for you if you could continue business operations with another hosting provider while things play out with Amazon. To that end, having backups (at minimum) that are accessible without Amazon's involvement seems prudent.

(I'd even argue that it's probably worth doing an assessment of spinning up your entire application on another host. If things did go sour with Amazon having backups is nice, but it would be nicer still if you could continue running your site. That may be pie-in-the-sky depending on how deeply integrated your application is to Amazon's platform, but it's at least worth discussing.)

  • When I was thinking about another account I was thinking about an account in another name using completely different billing, so if like you said for whatever reason Amazon throws us overboard we could set everything up again on that other account. I think we would have had to have pissed off Amazon pretty badly for them to stalk us across accounts. I don't think the line of work we're in would ever set us up for that, but it doesn't hurt to be cautious I suppose. – ChrisInCambo Nov 26 '14 at 4:50
  • @ChrisInCambo - It depends. If Amazon ended up with some kind injunction preventing your access to the service it probably wouldn't matter, from a legal perspective, if you were using a different account w/ different billing. That would be a Talk to a Lawyer(tm) moment, for sure. (I tend to think about worst-case scenarios and often rely on others to reign me back in and temper my visions of woe.) – Evan Anderson Nov 26 '14 at 4:52
  • 2
    If you need to shift providers, you might want to try fielding a Eucalyptus test lab in the event you have to move off AWS. Its goal is to be compatible. – Andrew Domaszek Nov 26 '14 at 5:04
  • @AndrewDomaszek - Thanks for that! I was trying to think of Eucalyptus' name but couldn't come up with it. – Evan Anderson Nov 26 '14 at 5:05
5

It looks like your threat model is pretty good.

AWS provides availability zones on EC2 instances (and related services) to help guard against this type of thing a little bit. Putting backups in another region is even better.

It's not so much about Amazon's immunity or non-immunity to damage, but about the concept of backups being separated by physical distance.

The threat model related to someone compromising your account is totally reasonable; people have been held for ransom that way in the past.

Personally I wouldn't move the snapshots. If you are using EC2 servers as anything but ephemeral nodes, you aren't using the full power of AWS. The point of a "cloud architecture" is that the servers can be zapped at any time. But, I would definitely apply this paradigm to actual backups (dumps of data, etc.).

Your alternative to putting backups into a separate account and probably a separate AWS region is much more expensive: an offsite data storage company. Those guys are usually rather more costly, but tend to make explicit statements about how safe your data is, in exchange.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.