This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

51
votes
10answers
37k views

How can I get the size of an Amazon S3 bucket?

I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data. The s3cmd tools provide a way to get the total file size using s3cmd ...
21
votes
6answers
20k views

Using RSYNC with Amazon S3

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools) Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
20
votes
4answers
16k views

How to move files between two S3 buckets with minimum cost?

I have millions of files in a Amazon S3 bucket and I'd like to move these files to other buckets and folders with minimum cost or no cost if possible. All buckets are in the same zone. How could I ...
19
votes
6answers
12k views

AWS S3 Bucket Backups

Is there any way to recover from accidental deletions of an Amazon S3 Bucket? We've got critical info in our buckets and I need to mitigate the risk of accidental or malicious deletions of the bucket ...
14
votes
11answers
5k views

S3-compatible Object Storage systems [closed]

Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system? What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 ...
11
votes
1answer
4k views

Duplicity and Amazon S3 Script

I'm looking to backup my linux server to Amazon S3 using duplicity. I found a great resource here that helped me get it setup and am using the basic script that was listed there for me and now copied ...
10
votes
3answers
3k views

Any need to make backup of data on Amazon S3?

I'm hosting 200 GB of product images at S3 (this is my primary file host). Do I need to back that data up somewhere else, or is S3 safe as it is? I have been experimenting with mounting the S3 ...
9
votes
9answers
5k views

How to store 3 million records in key value format?

We have to store basic information about 3 million products. Currently the info is one 180 mb CSV which gets updated quarterly. There will be about 30,000 queries per day, but the queries are just a ...
9
votes
3answers
2k views

What's the easiest way to back up EC2 instances automatically?

I'm worried I might log in one day and my fully configured instance has been removed. I would like to be able to back it up daily to S3 but frustratingly this is not a simple setting in the console. ...
9
votes
1answer
3k views

Amazon Cloudfront with S3. Access Denied

We're trying to distribute out S3 buckets via Cloudfront but for some reason the only response is an AccessDenied XML document like the following: <Error> ...
8
votes
2answers
6k views

Which is the fastest way to copy 400G of files from an ec2 elastic block store volume to s3?

I have to copy 400G of files from an elastic block store volume to an s3 bucket... Those are about 300k files of ~1Mb I've tried s3cmd and s3fuse, both of them are really, really slow.. s3cmd ran for ...
8
votes
3answers
3k views

Backing up data stored on Amazon S3

I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate ...
8
votes
3answers
2k views

Amazon S3 and Access-Control-Allow-Origin

Does Amazon S3 allow custom headers? Or am I out of luck? Access-Control-Allow-Origin: * Thanks in advance!
8
votes
2answers
4k views

EC2 - Shared Storage - S3FS or EBS?

Building my web service onto EC2 right now and have a single instance behind a load balancer. I will of course cater for multiple instances. My initial idea was to run all the instances a dumb ...
7
votes
6answers
1k views

Does a CDN still work even when my server is down?

I am a website owner planning to use Amazon's cloudfront S3. I have been reading all the stuffs about what a CDN can do but i still have an unanswered question. Does a CDN still work even when my ...
7
votes
5answers
3k views

What is a practical way to mirror an Amazon S3 bucket?

I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the ...
7
votes
5answers
9k views

How can one efficiently use S3 to back up files incrementally?

I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP. There look to be a few approaches. s3rsync ...
7
votes
2answers
8k views

Is mounting S3 buckets directly into EC2 instances safe?

as the title of my question says, I would like to know whether this is a good practice or not, I know I can mount it in an EC2 instance but I was wondering if, as I will be adding more instances to ...
7
votes
3answers
10k views

Why is uploading to S3 so slow?

I am using s3cmd to upload to S3: # s3cmd put 1gb.bin s3://my-bucket/1gb.bin 1gb.bin -> s3://my-bucket/1gb.bin [1 of 1] 366706688 of 1073741824 34% in 371s 963.22 kB/s I am uploading ...
7
votes
2answers
3k views

What's the best practice for taking MySQL dump, encrypting it and then pushing to s3?

This current project requires that the DB be dumped, encrypted and pushed to s3. I'm wondering what might be some "best practices" for such a task. As of now I'm using a pretty straight ahead method ...
7
votes
4answers
837 views

What is the cheapest non-colocation way to serve about 10 static files at a rate of 100 megabits per second to the web? [closed]

I've looked at Amazon S3 and it costs roughly $4746 per month for 100 megabits/s (which translates into 31,640 Gigabytes of data transferred. That's at a rate of $0.15 per gig.) I haven't found a ...
6
votes
5answers
7k views

s3cmd fails too many times

It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network: root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/ ...
6
votes
4answers
3k views

How to move S3 bucket to different location

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to ...
6
votes
1answer
1k views

AWS IAM user access issues for S3

I have the following IAM policy for a user { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1395161912000", "Effect": "Allow", "Action": [ "s3:ListBucket", ...
6
votes
6answers
2k views

Amazon EC2 & S3 costs - can they be tied to specific instances [closed]

I'd like to start using S3 and EC2 to host some of my company's simpler websites. I would like to be able to identify all of the costs associated with running each site (instance run-time costs + ...
6
votes
5answers
608 views

Does Amazon S3 make any sort of guarantees on your data?

I've read over the agreements several times, and I can't find anything that guarantees Amazon will not lose or corrupt your data. Do they make any such guarantees? Can I trust S3 as my long term ...
6
votes
3answers
3k views

Amazon S3 not sending Content-Type header

I have an application that downloads content from various sources. It relies on the "Content-Type" header being set on images. The majority of web-servers do this correctly but it appears Amazon S3 ...
6
votes
3answers
2k views

Amazon CloudFront and EC2: Global Load Balancing

We have an app that is going to store and serve up a decent amount of data in S3 to a global audience where latency should be minimized. So, we've been doing tests with Amazon CloudFront and have seen ...
6
votes
2answers
2k views

AWS S3 Backup Strategies - How should I approach backing up S3 buckets?

I am in the process of building a web-app with potential for a massive amount of storage requirements which can be satisfied by amazon S3. My main concern is the usage of API keys on the server, and ...
6
votes
4answers
3k views

MogileFS/GlusterFS/etc + Amazon EBS + Amazon EC2

I have a web application that serves binary files (images, etc). Our application runs on Amazon EC2. We were originally going to use Amazon S3 to store and serve these files, this is no longer an ...
6
votes
2answers
3k views

Can you restore a MySQL dump file from an S3 bucket to an RDS instance?

I'm investigating running regular MySQL dumps to an S3 bucket as part of a disaster recovery strategy (partly spurred by the current zero cost of inbound data transfer!). In the event of a disaster, I ...
5
votes
2answers
6k views

CNAME record for Amazon S3 — any drawbacks?

I'd like to make a CNAME record for my Amazon S3 domain, to have nicer URLs. There's not much discussion of this out there on the web, and nothing at all in the Amazon documentation (unless I'm ...
5
votes
1answer
2k views

No targets available when trying to set alias target from Route 53 to S3

I'm trying to setup a simple Amazon AWS S3 based website, as explained here. I've setup the S3 bucket (simples3websitetest.com), gave it the (hopefully) right permissions: { "Version": ...
5
votes
3answers
4k views

Forcing CloudFront to pass-through the latest HTML file from S3

Background I'm hosting a static site on S3, with CloudFront over the top. The issue I have is with my HTML files. According to CloudFront's FAQ: Amazon CloudFront uses these cache control ...
5
votes
4answers
9k views

How to schedule automatic (daily) snapshots of AWS EC2 Windows Instance?

I have some Windows servers hosted on Amazon EC2. Some run Windows Server 2003 and other run Windows Server 2008. These are EBS-backed instances. Most of the instances also have some additional ...
5
votes
2answers
2k views

Duplicity not writing to a pre-existing S3 bucket

I'm trying to backup a directory to a pre-existing Amazon S3 bucket using the following command: duplicity --no-encryption system/ s3+http://MY_BUCKET_NAME/backup However, I'm getting the following ...
5
votes
5answers
520 views

What sort of web server hardware do you use to handle 100 Mbps+ of static files?

I currently use Amazon S3 for much of my static file serving needs but my monthly bill is getting very expensive. I did some rough calculations using the logs and at peak times, my most expensive ...
5
votes
2answers
2k views

Webstats for large Amazon S3/Cloudfront logs?

I'm trying to find a way to parse our Amazon S3 access logs to get some webstats. I've been trying to use AWStats 7, but I got to the point of where after day 9 of a given month it can't process any ...
5
votes
2answers
238 views

Is there software that can help me easily back up my data from Amazon S3 to my premise?

So everyone backs their stuff up to the cloud. But I want to back up my stuff from the cloud. For a planned SaaS service, users are saving files containing their intellectual property and we're ...
5
votes
1answer
62 views

Deployment to thousands of Amazon S3 accounts?

We run a service whereby we provide the 'work' of a web (PHP)-based app and the images, JS, CSS etc are hosted on the clients' own Amazon S3 accounts. This is so that they get a consolidated bill for ...
4
votes
3answers
4k views

Uploading 200GB of files to S3

What's the best way to upload 200GB tar.gz files to S3 in Linux? On researching I found that S3 limit on objects has been increased to 5TB and came to know about the multi part upload mechanism to ...
4
votes
2answers
1k views

What design features make Joyent's ZFS and Amazon's EBS (S3) reliable?

I know this isn't exactly an apples to apples comparison, what I'm trying to evaluate is which one is safer - less likely to lose data. Joyent's SmartOS uses ZFS to store data whereas an EC2 machine ...
4
votes
1answer
112 views

Is Amazon s3 required if am using Cloudfront for files

I am planning to host a website in Amazon Ec2 instant. I got some basics from the docs. But I have few confusions. If am using cloudfront for CDN, do i really need to server files from s3? The site ...
4
votes
1answer
1k views

Why I'm getting MD5 Sums don't match! error?

I'm trying to upload my audit files to S3. Using s3cmd tools. I just run this command : sudo s3cmd put /var/log/audit/audit.log s3://audit-files But I get a error message like this : ...
4
votes
2answers
2k views

Snapshots == EBS backed instance, persistence, and S3 storage?

I want to clear up some confusion and clarify understanding on EC2 instances being EBS backed. If you start an AMI from a community image, configure your server as you want it configured, then use ...
4
votes
1answer
5k views

How to force s3fs mount on boot

I'm using s3fs 1.33 on Ubuntu 9.10 x86. I mount it manually with this command: sudo /usr/bin/s3fs mybucket /mnt/s3/mybucket -ouse_cache=/tmp How can I force the mount on boot? I guess that I have ...
4
votes
2answers
712 views

How long before an s3 bucket can be created with same name after deletion?

Title is quite explanatory, but I have just deleted an s3 bucket as it was in the wrong region and am wanting to recreate it in the correct region with same name as the just deleted one. Is there any ...
4
votes
2answers
643 views

Common storage for EC2 Instances

I wanted to attach a EC2 volume to multiple EC2 instance and create a common directory for all my scripts. I was not able to do this. Is it possible to mount some shared storage between all my EC2 ...
4
votes
1answer
692 views

Most efficient way to copy between S3 accounts

I am migrating data from one S3 account (billed to me) to another S3 account (billed to my company). How do I copy the data in the easiest, fastest, and most cost-effective way? There are some ...
4
votes
2answers
539 views

EC2 persistence of machine

I notice that EBS-backed AMIs are much like a VMWare instances -- I can stop them and also persist them to disk, and all this is done relatively quickly. However, I believe that S3 backed machines ...