This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

42
votes
9answers
27k views

How can I get the size of an Amazon S3 bucket?

I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data. The s3cmd tools provide a way to get the total file size using s3cmd ...
21
votes
6answers
18k views

Using RSYNC with Amazon S3

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools) Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
19
votes
4answers
13k views

How to move files between two S3 buckets with minimum cost?

I have millions of files in a Amazon S3 bucket and I'd like to move these files to other buckets and folders with minimum cost or no cost if possible. All buckets are in the same zone. How could I ...
16
votes
6answers
10k views

AWS S3 Bucket Backups

Is there any way to recover from accidental deletions of an Amazon S3 Bucket? We've got critical info in our buckets and I need to mitigate the risk of accidental or malicious deletions of the bucket ...
12
votes
11answers
4k views

S3-compatible Object Storage systems [closed]

Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system? What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 ...
11
votes
1answer
4k views

Duplicity and Amazon S3 Script

I'm looking to backup my linux server to Amazon S3 using duplicity. I found a great resource here that helped me get it setup and am using the basic script that was listed there for me and now copied ...
10
votes
3answers
3k views

Any need to make backup of data on Amazon S3?

I'm hosting 200 GB of product images at S3 (this is my primary file host). Do I need to back that data up somewhere else, or is S3 safe as it is? I have been experimenting with mounting the S3 ...
9
votes
9answers
4k views

How to store 3 million records in key value format?

We have to store basic information about 3 million products. Currently the info is one 180 mb CSV which gets updated quarterly. There will be about 30,000 queries per day, but the queries are just a ...
9
votes
8answers
1k views

S3 Interfaces for Backup, etc [closed]

Since none of the S3 "file systems" are compatible right now, what s3 gateway tool do you recommend for mounting a remote S3 bucket for backups? I've been using S3FS, but I've heard a lot of people ...
9
votes
3answers
2k views

What's the easiest way to back up EC2 instances automatically?

I'm worried I might log in one day and my fully configured instance has been removed. I would like to be able to back it up daily to S3 but frustratingly this is not a simple setting in the console. ...
8
votes
2answers
4k views

EC2 - Shared Storage - S3FS or EBS?

Building my web service onto EC2 right now and have a single instance behind a load balancer. I will of course cater for multiple instances. My initial idea was to run all the instances a dumb ...
7
votes
2answers
7k views

Is mounting S3 buckets directly into EC2 instances safe?

as the title of my question says, I would like to know whether this is a good practice or not, I know I can mount it in an EC2 instance but I was wondering if, as I will be adding more instances to ...
7
votes
3answers
2k views

Backing up data stored on Amazon S3

I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate ...
7
votes
2answers
2k views

What's the best practice for taking MySQL dump, encrypting it and then pushing to s3?

This current project requires that the DB be dumped, encrypted and pushed to s3. I'm wondering what might be some "best practices" for such a task. As of now I'm using a pretty straight ahead method ...
7
votes
3answers
2k views

Amazon S3 and Access-Control-Allow-Origin

Does Amazon S3 allow custom headers? Or am I out of luck? Access-Control-Allow-Origin: * Thanks in advance!
6
votes
6answers
1k views

Does a CDN still work even when my server is down?

I am a website owner planning to use Amazon's cloudfront S3. I have been reading all the stuffs about what a CDN can do but i still have an unanswered question. Does a CDN still work even when my ...
6
votes
5answers
6k views

s3cmd fails too many times

It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network: root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/ ...
6
votes
4answers
6k views

How can one efficiently use S3 to back up files incrementally?

I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP. There look to be a few approaches. s3rsync ...
6
votes
2answers
4k views

Which is the fastest way to copy 400G of files from an ec2 elastic block store volume to s3?

I have to copy 400G of files from an elastic block store volume to an s3 bucket... Those are about 300k files of ~1Mb I've tried s3cmd and s3fuse, both of them are really, really slow.. s3cmd ran for ...
6
votes
4answers
2k views

What is a practical way to mirror an Amazon S3 bucket?

I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the ...
6
votes
6answers
2k views

Amazon EC2 & S3 costs - can they be tied to specific instances [closed]

I'd like to start using S3 and EC2 to host some of my company's simpler websites. I would like to be able to identify all of the costs associated with running each site (instance run-time costs + ...
6
votes
5answers
601 views

Does Amazon S3 make any sort of guarantees on your data?

I've read over the agreements several times, and I can't find anything that guarantees Amazon will not lose or corrupt your data. Do they make any such guarantees? Can I trust S3 as my long term ...
6
votes
3answers
2k views

Amazon CloudFront and EC2: Global Load Balancing

We have an app that is going to store and serve up a decent amount of data in S3 to a global audience where latency should be minimized. So, we've been doing tests with Amazon CloudFront and have seen ...
6
votes
2answers
2k views

AWS S3 Backup Strategies - How should I approach backing up S3 buckets?

I am in the process of building a web-app with potential for a massive amount of storage requirements which can be satisfied by amazon S3. My main concern is the usage of API keys on the server, and ...
6
votes
4answers
794 views

What is the cheapest non-colocation way to serve about 10 static files at a rate of 100 megabits per second to the web? [closed]

I've looked at Amazon S3 and it costs roughly $4746 per month for 100 megabits/s (which translates into 31,640 Gigabytes of data transferred. That's at a rate of $0.15 per gig.) I haven't found a ...
6
votes
3answers
3k views

Amazon S3 not sending Content-Type header

I have an application that downloads content from various sources. It relies on the "Content-Type" header being set on images. The majority of web-servers do this correctly but it appears Amazon S3 ...
6
votes
4answers
3k views

MogileFS/GlusterFS/etc + Amazon EBS + Amazon EC2

I have a web application that serves binary files (images, etc). Our application runs on Amazon EC2. We were originally going to use Amazon S3 to store and serve these files, this is no longer an ...
5
votes
2answers
5k views

CNAME record for Amazon S3 — any drawbacks?

I'd like to make a CNAME record for my Amazon S3 domain, to have nicer URLs. There's not much discussion of this out there on the web, and nothing at all in the Amazon documentation (unless I'm ...
5
votes
3answers
3k views

Forcing CloudFront to pass-through the latest HTML file from S3

Background I'm hosting a static site on S3, with CloudFront over the top. The issue I have is with my HTML files. According to CloudFront's FAQ: Amazon CloudFront uses these cache control ...
5
votes
4answers
3k views

How to move S3 bucket to different location

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to ...
5
votes
5answers
499 views

What sort of web server hardware do you use to handle 100 Mbps+ of static files?

I currently use Amazon S3 for much of my static file serving needs but my monthly bill is getting very expensive. I did some rough calculations using the logs and at peak times, my most expensive ...
5
votes
2answers
3k views

Can you restore a MySQL dump file from an S3 bucket to an RDS instance?

I'm investigating running regular MySQL dumps to an S3 bucket as part of a disaster recovery strategy (partly spurred by the current zero cost of inbound data transfer!). In the event of a disaster, I ...
5
votes
2answers
228 views

Is there software that can help me easily back up my data from Amazon S3 to my premise?

So everyone backs their stuff up to the cloud. But I want to back up my stuff from the cloud. For a planned SaaS service, users are saving files containing their intellectual property and we're ...
5
votes
1answer
60 views

Deployment to thousands of Amazon S3 accounts?

We run a service whereby we provide the 'work' of a web (PHP)-based app and the images, JS, CSS etc are hosted on the clients' own Amazon S3 accounts. This is so that they get a consolidated bill for ...
4
votes
3answers
3k views

Uploading 200GB of files to S3

What's the best way to upload 200GB tar.gz files to S3 in Linux? On researching I found that S3 limit on objects has been increased to 5TB and came to know about the multi part upload mechanism to ...
4
votes
3answers
7k views

Why is uploading to S3 so slow?

I am using s3cmd to upload to S3: # s3cmd put 1gb.bin s3://my-bucket/1gb.bin 1gb.bin -> s3://my-bucket/1gb.bin [1 of 1] 366706688 of 1073741824 34% in 371s 963.22 kB/s I am uploading ...
4
votes
2answers
1k views

What design features make Joyent's ZFS and Amazon's EBS (S3) reliable?

I know this isn't exactly an apples to apples comparison, what I'm trying to evaluate is which one is safer - less likely to lose data. Joyent's SmartOS uses ZFS to store data whereas an EC2 machine ...
4
votes
1answer
100 views

Is Amazon s3 required if am using Cloudfront for files

I am planning to host a website in Amazon Ec2 instant. I got some basics from the docs. But I have few confusions. If am using cloudfront for CDN, do i really need to server files from s3? The site ...
4
votes
4answers
6k views

How to schedule automatic (daily) snapshots of AWS EC2 Windows Instance?

I have some Windows servers hosted on Amazon EC2. Some run Windows Server 2003 and other run Windows Server 2008. These are EBS-backed instances. Most of the instances also have some additional ...
4
votes
2answers
1k views

Duplicity not writing to a pre-existing S3 bucket

I'm trying to backup a directory to a pre-existing Amazon S3 bucket using the following command: duplicity --no-encryption system/ s3+http://MY_BUCKET_NAME/backup However, I'm getting the following ...
4
votes
2answers
2k views

Snapshots == EBS backed instance, persistence, and S3 storage?

I want to clear up some confusion and clarify understanding on EC2 instances being EBS backed. If you start an AMI from a community image, configure your server as you want it configured, then use ...
4
votes
2answers
589 views

Common storage for EC2 Instances

I wanted to attach a EC2 volume to multiple EC2 instance and create a common directory for all my scripts. I was not able to do this. Is it possible to mount some shared storage between all my EC2 ...
4
votes
1answer
691 views

Most efficient way to copy between S3 accounts

I am migrating data from one S3 account (billed to me) to another S3 account (billed to my company). How do I copy the data in the easiest, fastest, and most cost-effective way? There are some ...
4
votes
2answers
537 views

EC2 persistence of machine

I notice that EBS-backed AMIs are much like a VMWare instances -- I can stop them and also persist them to disk, and all this is done relatively quickly. However, I believe that S3 backed machines ...
4
votes
2answers
1k views

Webstats for large Amazon S3/Cloudfront logs?

I'm trying to find a way to parse our Amazon S3 access logs to get some webstats. I've been trying to use AWStats 7, but I got to the point of where after day 9 of a given month it can't process any ...
4
votes
0answers
181 views

Amazon S3 appears to be blocking cloudflare IP addresses. How do I fix it?

I have static content that is being served by Cloudflare. Cloudflare points to Amazon S3 to pick up the static content and serves it via a CNAME (cdn.mydomain.com). The bucket is setup properly and ...
4
votes
3answers
273 views

AMIs in Amazon EC2

I really like the Amazon EC2 environment, and thought I'll spend a bit of time playing around with various types of public (Windows!) AMI servers. But testing has been a bit, well, questionable. Some ...
3
votes
3answers
727 views

Precautions when using Amazon's S3

I am in the process of building a website that will distribute a large amount (>150GB) of small files (~25Mb Avg.) to a large number of people, since both bandwidth and disk space are expensive with ...
3
votes
2answers
195 views

Storing and serving files securely for multiple clients

We are working on a web app, where (among other features) our users can upload their files. However we can't store these files on our VPS because storage space is limited, so we decided to go with S3. ...
3
votes
2answers
2k views

Amazon pricing - transfer between EC2, EBS and S3 [closed]

I have read quite a bit on amazon and the pricing details for EC2, EBS and S3 but i still have a few questions: If i understand correctly, traffic between EC2 instances isn't billed. Is the data ...