We are deploying a number of servers at AWS and need to work-out a general-purpose backup strategy. The requirements are:

  1. Retain the ability to recover full systems or individual files or subdirectories going back up to N weeks (with N varying per system).
  2. Be able to exclude certain sub-trees (or mount-points).
  3. Store the backups on S3 (or even Glacier).
  4. Minimize costs and times -- use incremental backups to avoid storing too many copies of unchanged files.

EC2 snapshots seems like an obvious solution, but recovering individual files while the originating system is still running is, apparently, not always possible (the dreadful "Marketplace product code" error message). Also, it is only possible to dump the entire filesystem (volume) -- without a way to exclude anything.

That leaves the "traditional" options of Amanda and Duplicity, which nowadays both have S3 as the storage option.

Am I right with my assessment of limitations of the EC2 snapshots? Any comments comparing usage of Amanda vs. Duplicity at AWS? Thank you!

closed as off-topic by mdpc, MichelZ, dyasny, Katherine Villyard, HopelessN00b Jun 25 '14 at 19:33

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they tend to become obsolete quickly. Instead, describe your situation and the specific problem you're trying to solve." – mdpc, MichelZ, dyasny, Katherine Villyard, HopelessN00b
If this question can be reworded to fit the rules in the help center, please edit the question.

  • You probably ought to think about whether or not you really want backups of your AWS infrastructure to be stored in AWS. – EEAA Jun 24 '14 at 2:13

For full system backups in EC2, you should go with EBS Snapshots. For data backup in S3/Glacier I would go with S3cmd in Linux systems and Cloudberry Backup tools in Windows (Not free, but worths each dollar).

S3cmd is a simple command line tool you can use in your standard backup scripts to push a copy into a S3 bucket. For Glacier rotations, use S3 lifecycle rules. Works like a charm.

In Windows, I have not been able to find a decent free backup tool for S3, but Cloudberry is just great. It allows you to configure hundreds of options (Versioning, file exclusions, multiple storage backends...). Also, no issue with it up to date.

Bonus Track: For SQL Server specific S3 backups, I would use the Keymetric SQL Backup Master solution instead of Cloudberry's one. It's simpler, effective and cheap.

  • I didn't downvote, but you are really trusting Amazon with your data. – Jacob Jun 24 '14 at 13:06
  • Is your strategy capable of meeting the requirements I listed? We aren't using Windows here, but any plan for backing up MySQL databases may be of interest. – Mikhail T. Jun 24 '14 at 14:07
  • IMHO, S3cmd sync command + --exclude parameters for subtree exclusion + S3 bucket versioning should give you the functionality you intend to achieve. – ma.tome Jun 24 '14 at 15:05
  • @Jacob, I understood from the question that the objective was S3 backed backups. – ma.tome Jun 24 '14 at 15:07
  • The backups shall end up stored on S3, correct. But which tool would create them -- "ec2 create-snapshot" or something else -- is my question... – Mikhail T. Jun 24 '14 at 15:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.