This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

129
votes
22answers
93k views

How can I get the size of an Amazon S3 bucket?

I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data. The s3cmd tools provide a way to get the total file size using s3cmd ...
26
votes
4answers
23k views

How to move files between two S3 buckets with minimum cost?

I have millions of files in a Amazon S3 bucket and I'd like to move these files to other buckets and folders with minimum cost or no cost if possible. All buckets are in the same zone. How could I ...
25
votes
6answers
27k views

How do I back up an AWS S3 Bucket without versioning the source bucket [closed]

Is there any way to recover from accidental deletions of an Amazon S3 Bucket? We've got critical info in our buckets and I need to mitigate the risk of accidental or malicious deletions of the bucket ...
24
votes
7answers
24k views

How can one efficiently use S3 to back up files incrementally?

I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP. There look to be a few approaches. s3rsync ...
24
votes
7answers
32k views

Using RSYNC with Amazon S3

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools) Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
24
votes
3answers
42k views

What is causing Access Denied when using the aws cli to download from Amazon S3?

I'm really flailing around in AWS trying to figure out what I'm missing here. I'd like to make it so that an IAM user can download files from an S3 bucket - without just making the files totally ...
18
votes
2answers
16k views

Amazon Cloudfront with S3. Access Denied

We're trying to distribute out S3 buckets via Cloudfront but for some reason the only response is an AccessDenied XML document like the following: <Error> <Code>AccessDenied</Code&...
17
votes
6answers
14k views

Which is the fastest way to copy 400G of files from an ec2 elastic block store volume to s3?

I have to copy 400G of files from an elastic block store volume to an s3 bucket... Those are about 300k files of ~1Mb I've tried s3cmd and s3fuse, both of them are really, really slow.. s3cmd ran for ...
17
votes
1answer
9k views

No targets available when trying to set alias target from Route 53 to S3

I'm trying to setup a simple Amazon AWS S3 based website, as explained here. I've setup the S3 bucket (simples3websitetest.com), gave it the (hopefully) right permissions: { "Version": "2012-10-...
15
votes
5answers
2k views

Software for using Amazon S3 as backup service [closed]

What software do you recommend for using Amazon S3 as a placeholder for daily backups? I need it to be fully automated, and easy to restore data.
14
votes
3answers
6k views

Backing up data stored on Amazon S3

I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate ...
14
votes
10answers
9k views

S3-compatible Object Storage systems [closed]

Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system? What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 ...
13
votes
5answers
8k views

How to move S3 bucket to different location

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to ...
13
votes
1answer
6k views

AWS IAM user access issues for S3

I have the following IAM policy for a user { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1395161912000", "Effect": "Allow", "Action": [ "s3:ListBucket", ...
12
votes
1answer
5k views

Duplicity and Amazon S3 Script

I'm looking to backup my linux server to Amazon S3 using duplicity. I found a great resource here that helped me get it setup and am using the basic script that was listed there for me and now copied ...
12
votes
3answers
4k views

Any need to make backup of data on Amazon S3?

I'm hosting 200 GB of product images at S3 (this is my primary file host). Do I need to back that data up somewhere else, or is S3 safe as it is? I have been experimenting with mounting the S3 ...
10
votes
9answers
7k views

How to store 3 million records in key value format?

We have to store basic information about 3 million products. Currently the info is one 180 mb CSV which gets updated quarterly. There will be about 30,000 queries per day, but the queries are just a ...
10
votes
8answers
1k views

S3 Interfaces for Backup, etc [closed]

Since none of the S3 "file systems" are compatible right now, what s3 gateway tool do you recommend for mounting a remote S3 bucket for backups? I've been using S3FS, but I've heard a lot of people ...
10
votes
1answer
10k views

Setting Access-Control-Allow-Origin on Cloudfront

I am having problems serving static assets to Firefox using AWS Cloudfront. Chrome works perfect, but Firefox is returning a CORS error. If I execute curl , I get: HTTP/1.1 200 OK Content-Type: ...
10
votes
3answers
3k views

Amazon S3 and Access-Control-Allow-Origin

Does Amazon S3 allow custom headers? Or am I out of luck? Access-Control-Allow-Origin: * Thanks in advance!
9
votes
3answers
20k views

Why is uploading to S3 so slow?

I am using s3cmd to upload to S3: # s3cmd put 1gb.bin s3://my-bucket/1gb.bin 1gb.bin -> s3://my-bucket/1gb.bin [1 of 1] 366706688 of 1073741824 34% in 371s 963.22 kB/s I am uploading ...
9
votes
3answers
4k views

How long before an s3 bucket can be created with same name after deletion?

Title is quite explanatory, but I have just deleted an s3 bucket as it was in the wrong region and am wanting to recreate it in the correct region with same name as the just deleted one. Is there any ...
9
votes
3answers
2k views

What's the easiest way to back up EC2 instances automatically?

I'm worried I might log in one day and my fully configured instance has been removed. I would like to be able to back it up daily to S3 but frustratingly this is not a simple setting in the console. ...
8
votes
6answers
2k views

Does a CDN still work even when my server is down?

I am a website owner planning to use Amazon's cloudfront S3. I have been reading all the stuffs about what a CDN can do but i still have an unanswered question. Does a CDN still work even when my ...
8
votes
2answers
1k views

Access Amazon S3 Static Website Through IPv6

I host a static website on Amazon S3 with DNS hosted on Amazon Route 53. I'd like my site to be accessible via IPv4 and IPv6 (because it's about time). When I use the Route 53 AWS Management Console ...
8
votes
6answers
24k views

How to schedule automatic (daily) snapshots of AWS EC2 Windows Instance?

I have some Windows servers hosted on Amazon EC2. Some run Windows Server 2003 and other run Windows Server 2008. These are EBS-backed instances. Most of the instances also have some additional EBS-...
8
votes
3answers
5k views

Copy directory structure intact to AWS S3 bucket

I want to use the AWS S3 cli to copy a full directory structure to an S3 bucket. So far, everything I've tried copies the files to the bucket, but the directory structure is collapsed. (to say it ...
8
votes
3answers
4k views

Amazon S3 not sending Content-Type header

I have an application that downloads content from various sources. It relies on the "Content-Type" header being set on images. The majority of web-servers do this correctly but it appears Amazon S3 ...
8
votes
3answers
4k views

What's the best practice for taking MySQL dump, encrypting it and then pushing to s3?

This current project requires that the DB be dumped, encrypted and pushed to s3. I'm wondering what might be some "best practices" for such a task. As of now I'm using a pretty straight ahead method ...
8
votes
2answers
4k views

EC2 - Shared Storage - S3FS or EBS?

Building my web service onto EC2 right now and have a single instance behind a load balancer. I will of course cater for multiple instances. My initial idea was to run all the instances a dumb slaves,...
7
votes
6answers
10k views

s3cmd fails too many times

It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network: root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/ ...
7
votes
5answers
6k views

Automatically delete old items from s3 bucket

Is there an easy way to set up a bucket in s3 to automatically delete files older than x days?
7
votes
4answers
9k views

Accessing Amazon S3 from a private VPC subnet

If I have a VPC running and some servers located in the private portion of that network that do backend processing by downloading files from amazon s3, can I access S3 internally to get at those files?...
7
votes
2answers
12k views

Is mounting S3 buckets directly into EC2 instances safe?

as the title of my question says, I would like to know whether this is a good practice or not, I know I can mount it in an EC2 instance but I was wondering if, as I will be adding more instances to ...
7
votes
6answers
4k views

What is a practical way to mirror an Amazon S3 bucket?

I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the ...
7
votes
3answers
8k views

Forcing CloudFront to pass-through the latest HTML file from S3

Background I'm hosting a static site on S3, with CloudFront over the top. The issue I have is with my HTML files. According to CloudFront's FAQ: Amazon CloudFront uses these cache control ...
7
votes
2answers
2k views

How can I reuse existing resources in CloudFormation?

I've have S3 bucket as resource in my CloudFormation template. With DeletionPolicy set to Retain. This works as expected, when deleting the stack, it does indeed retain the bucket. However, when I ...
7
votes
4answers
1k views

What is the cheapest non-colocation way to serve about 10 static files at a rate of 100 megabits per second to the web? [closed]

I've looked at Amazon S3 and it costs roughly $4746 per month for 100 megabits/s (which translates into 31,640 Gigabytes of data transferred. That's at a rate of $0.15 per gig.) I haven't found a ...
6
votes
3answers
6k views

What is the difference between Buckets and Folders in Amazon S3?

What is the difference between Buckets and Folders in Amazon S3 ? Is such a thing like Folder exist in Amazon S3 ? or only the S3 clients present Folders to us for better handling ?
6
votes
2answers
1k views

Storing and serving files securely for multiple clients

We are working on a web app, where (among other features) our users can upload their files. However we can't store these files on our VPS because storage space is limited, so we decided to go with S3. ...
6
votes
3answers
247 views

Likely causes of NTPD dying unexpectedly and solutions

On a web application which uses s3 for physical document storage, we are experiencing issues with NTP continuously dying. This seems to happen roughly once or twice a day. There is very little ...
6
votes
6answers
2k views

Amazon EC2 & S3 costs - can they be tied to specific instances [closed]

I'd like to start using S3 and EC2 to host some of my company's simpler websites. I would like to be able to identify all of the costs associated with running each site (instance run-time costs + ...
6
votes
5answers
628 views

Does Amazon S3 make any sort of guarantees on your data?

I've read over the agreements several times, and I can't find anything that guarantees Amazon will not lose or corrupt your data. Do they make any such guarantees? Can I trust S3 as my long term ...
6
votes
2answers
3k views

AWS S3 Backup Strategies - How should I approach backing up S3 buckets?

I am in the process of building a web-app with potential for a massive amount of storage requirements which can be satisfied by amazon S3. My main concern is the usage of API keys on the server, and ...
6
votes
1answer
6k views

Give EC2 IAM role read access to S3 bucket

I have an AWS Elastic Beanstalk Rails app that I am configuring via the config script to pull some files from an S3 bucket. When I start up the application, I keep receiving the following error in the ...
6
votes
3answers
2k views

Amazon CloudFront and EC2: Global Load Balancing

We have an app that is going to store and serve up a decent amount of data in S3 to a global audience where latency should be minimized. So, we've been doing tests with Amazon CloudFront and have seen ...
6
votes
4answers
3k views

MogileFS/GlusterFS/etc + Amazon EBS + Amazon EC2

I have a web application that serves binary files (images, etc). Our application runs on Amazon EC2. We were originally going to use Amazon S3 to store and serve these files, this is no longer an ...
6
votes
2answers
5k views

Can you restore a MySQL dump file from an S3 bucket to an RDS instance?

I'm investigating running regular MySQL dumps to an S3 bucket as part of a disaster recovery strategy (partly spurred by the current zero cost of inbound data transfer!). In the event of a disaster, I ...
5
votes
2answers
9k views

Why S3 website redirect location is not followed by CloudFront?

I have a website hosted on Amazon S3. It is the new version of an old website hosted on WordPress. I have set up some files with the metadata Website Redirect Locationto handle old location and ...
5
votes
2answers
6k views

CNAME record for Amazon S3 — any drawbacks?

I'd like to make a CNAME record for my Amazon S3 domain, to have nicer URLs. There's not much discussion of this out there on the web, and nothing at all in the Amazon documentation (unless I'm ...