20

I'm hosting 200 GB of product images at S3 (this is my primary file host).

Do I need to back that data up somewhere else, or is S3 safe as it is?

I have been experimenting with mounting the S3 bucket to a EC2 instance, and then making a nightly rsync backup. The problem is that it's about 3 million files, so it takes a while to generate the different rsync needs. The backup actually takes about 3 days to complete.

Any ideas how to do this better? (if it's even necessary?)

5 Answers 5

7

I've been doing research on this, funny enough.

Your backups to S3 can fail depending on your region because of eventual consistency; the basic warning is that if you do this enough, at some point you'll have errors opening or finding files as the filesystem magic in the background of Amazon syncs among servers, so your backups may not be reliable.

As for whether you need to save them another way, this depends on your risk management. Do you trust Amazon to hold your data?

It's possible they may lose something or have a larger failure of their storage system; they no doubt have clauses in their contracts specifying that if they lose your data, that's your problem. Not theirs. Also, seeing as your data is housed somewhere else, you don't know what they will do with it; law enforcement want your data? You may not even know someone else accessed it.

Do you trust it? If the data isn't key to your business and you're willing to accept this risk, then there's no need to download it to offsite-storage. If you are not willing to risk that your data will be safe in Amazon's storage servers out there, you should make arrangements to periodically dump it to your own storage.

In other words I don't think there is a straight answer to this as it depends on your risk tolerance and business needs. Many people wouldn't completely trust their income solely on storage with the cloud, personally I feel a little wary of that...

To do this better, in discussions and research, another approach to consider is creating an EBS volume large enough to store the data, attach it to the EC2 instance, save your data there, then you can unmount the volume and save that data to S3. I'm in the middle of researching whether this would be done as saving the volume file itself to S3 or the contents...but then you can delete the EBS instance when done to save storage costs.

EDIT I see in re-reading that you're saving FROM S3 TO the EC2 instance, not vice-versa (although I don't know if the eventual consistency issue could still cause problems there). You're trying to save data to an EC2 instance as backup? I would think that cost-wise that's not a sound tactic; it may be cheaper to back things up to a local drive when you factor in long-term storage of that kind of data, along with VM time. With drive costs you could copy data down to a local disk as a backup.

I still would keep the warnings about trusting Amazon and their storage. If you want to keep everything in Amazon S3 but have more redundancy, duplicate your S3 buckets across regions, and if they have an outage affecting one region it shouldn't knock out all of them. You'd hope. Anything is possible though.

It comes down to how much you value your data, how much you're willing to pay for it and how much risk you want to tolerate.

4
  • Thanks for your answer, but I think you misunderstood. I'm using S3 as my primary storage (I'm hosting the files directly from there as a cdn). So my question was if the files are safe there, or if I need to make a backup of them somewere (might be to another S3 bucker or and EC2 instance)?
    – Chrille
    Feb 9, 2012 at 15:39
  • Added on an edit. Feb 9, 2012 at 15:54
  • Hmm, yeah maybe a local backup would be safer. Do you know if S3 and EC2 shares the same hardware - if it fails, both S3 and my EC2 backup would be gone? My two reasons for making a backup to EC2 is: (1) There is no transfer charge between EC2 and S3. It would cost pretty much to copy 200 GB locally every week or so. (2) in case of a disaster at S3, I could quite quickly convert my EC2 instance to share the images from the backup instead. But I get that a local backup has many advantages too...
    – Chrille
    Feb 9, 2012 at 17:42
  • I honestly don't know how Amazon's hardware is set up in the background; even if I knew, there's no guarantee they won't change it in the future. Feb 9, 2012 at 17:55
4

I've used s3cmd's s3cmd sync to do this. It's a bit rsync-like in it's operation, and can push and pull whole directories between S3 and another linux system of your choice.

I don't see any reason why you couldn't s3cmd sync to a running EC2 instance, or even your own developer workstation (or a storage server).

You might want to set up a VPC instance, and then you could assign a small node inside your VPC the role of backup server, and give it both an IP inside Amazon's network, as well as inside of your local subnet.

3
  • My concern, which he'd have to decide with his own financials, is the cost of maintaining an EC2 instance and the EBS space to store that kind of data compared to saving it to a local external hard drive for a couple hundred bucks instead. If money allows for the transfer fees I'd just download it to a local drive and keep that in sync periodically (which was also part of your solution given.) Feb 9, 2012 at 16:02
  • Amazon's EC2 isn't cheap by any measure of the term, especially if you want to do enterprise-level, or anything other than the simple stuff. If you don't like that, then perhaps it's not for you. Feb 9, 2012 at 16:03
  • @BartSilverstrim: isn't the transfer within AWS free? If so, it might be cheaper for me to copy to EC2 than locally. I have a EC2 instance running 24/7 anyway, so it's just the EBS space that would cost.
    – Chrille
    Feb 9, 2012 at 17:47
3

My advice is your data is your responsibility, not Amazon's. If losing the data is not such a big deal, then don't do your own backup. If it is, then take your own backup to (at the very least) a cheap JBOD (and verify regularly) as I do.

You'll find out how much responsibility Amazon is willing to assume for your data, the day they lose it.

0

If you can afford it (as I do this) is have all my data stored on my server, but pulling it from Amazon s3. So if Amazon goes down for any reason (touch wood) I can simply pull all my data instantly right from my server. From my server I make monthly backups to my local drive. As my website is over 2TB in site.

1
  • I find it a bit unclear what your recommendation is. What is your server and where is it located?
    – kasperd
    Oct 24, 2016 at 20:18
0

Although this is an old thread, it's the first thing that comes up when Googling S3 backup, so I thought I'd add to it...

Doing some research on this myself, I discovered Rclone https://rclone.org/ - it is rsync-ish software designed to copy between cloud file storage services and supports most of them. No affiliation and I haven't used it yet so I can't say if it's good or bad, but I thought it might help someone.

It seems to me that there is opportunity for a hosted service that does 'offsite' backups of cloud-hosted files (S3, Google Storage, Rackspace Cloud Files, etc)....

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .