We run a dozen or so Ubuntu Linux webserver production instances on Amazon VPC. The instances are bootstrapped and managed via Puppet. Most management is done via the AWS Console.
Our AWS credentials are pretty secure. The master-account is hardly ever needed, has a strong password and 2-factor auth. A few trusted admins have access to most services via their own IAM accounts, also with strong passwords and 2-factor auth. A few IAM accounts have very limited access for specific purposes, such as writing files to S3. Access by other employees to any high-level credentials is very limited. Overall, the chance of someone gaining access to the Console or API's seems low.
The recent Code Spaces debacle, where someone gained high-level access to their AWS Console and deleted instances, volumes and EBS Snapshots, effectively making it impossible for Code Spaces to recover their business, got me to investigate methods for backing up our data off-line/offsite (i.e. out of reach of our main AWS account).
How can I ensure our customer data is safe from being wiped out by someone who gains access to our AWS credentials, or by some disaster at AWS? Should be automatic, stable and reasonably priced.
I can't seem to be able to find an 'easy' way, after searching for a few hours. Copying EBS snapshots to another AWS account doesn't seem possible. I can't export EBS snapshots to S3 objects. I could rsync all important data by pulling from a third-party server but I'd need to script it to handle things like varying numbers of servers, retention, error-handling, etc. Seems like a lot of work. I found no ready-to-go software for this.
Our current backup strategy consists of nightly automated EBS Snapshots of all volumes, as well as uploading compressed MySQLdumps to S3. All source code and Puppet code is deployed from external version control, but our customers' files and MySQL databases are only stored on the EBS volumes and their snapshots, i.e. insider the AWS ecosystem.