2

We are deploying a number of servers at AWS and need to work-out a general-purpose backup strategy. The requirements are:

  1. Retain the ability to recover full systems or individual files or subdirectories going back up to N weeks (with N varying per system).
  2. Be able to exclude certain sub-trees (or mount-points).
  3. Store the backups on S3 (or even Glacier).
  4. Minimize costs and times -- use incremental backups to avoid storing too many copies of unchanged files.

EC2 snapshots seems like an obvious solution, but recovering individual files while the originating system is still running is, apparently, not always possible (the dreadful "Marketplace product code" error message). Also, it is only possible to dump the entire filesystem (volume) -- without a way to exclude anything.

That leaves the "traditional" options of Amanda and Duplicity, which nowadays both have S3 as the storage option.

Am I right with my assessment of limitations of the EC2 snapshots? Any comments comparing usage of Amanda vs. Duplicity at AWS? Thank you!

Improve this question
1
  • You probably ought to think about whether or not you really want backups of your AWS infrastructure to be stored in AWS.
    – EEAA
    Jun 24, 2014 at 2:13

1 Answer 1

Reset to default
1

For full system backups in EC2, you should go with EBS Snapshots. For data backup in S3/Glacier I would go with S3cmd in Linux systems and Cloudberry Backup tools in Windows (Not free, but worths each dollar).

S3cmd is a simple command line tool you can use in your standard backup scripts to push a copy into a S3 bucket. For Glacier rotations, use S3 lifecycle rules. Works like a charm.

In Windows, I have not been able to find a decent free backup tool for S3, but Cloudberry is just great. It allows you to configure hundreds of options (Versioning, file exclusions, multiple storage backends...). Also, no issue with it up to date.

Bonus Track: For SQL Server specific S3 backups, I would use the Keymetric SQL Backup Master solution instead of Cloudberry's one. It's simpler, effective and cheap.

Improve this answer
7
  • I didn't downvote, but you are really trusting Amazon with your data.
    – Jacob
    Jun 24, 2014 at 13:06
  • Is your strategy capable of meeting the requirements I listed? We aren't using Windows here, but any plan for backing up MySQL databases may be of interest.
    – Mikhail T.
    Jun 24, 2014 at 14:07
  • IMHO, S3cmd sync command + --exclude parameters for subtree exclusion + S3 bucket versioning should give you the functionality you intend to achieve.
    – ma.tome
    Jun 24, 2014 at 15:05
  • @Jacob, I understood from the question that the objective was S3 backed backups.
    – ma.tome
    Jun 24, 2014 at 15:07
  • The backups shall end up stored on S3, correct. But which tool would create them -- "ec2 create-snapshot" or something else -- is my question...
    – Mikhail T.
    Jun 24, 2014 at 15:50

Not the answer you're looking for? Browse other questions tagged .