This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

2
votes
3answers
961 views

Should I use ec2 as a file server?

I need to be able to share user uploaded content across multiple EC2 application servers. I have looked at rsync, mounted NFS, and S3 as potential options of being able to share this data in almost ...
2
votes
1answer
235 views

How do I implement IP-based export restrictions for my data stored on Amazon S3?

A little background: Corporate wants to make our downloads unavailable to U.S. export-restricted countries like North Korea, Iran, Syria, etc. Restriction by IP-block is considered good enough. (Yes, ...
2
votes
2answers
4k views

How to calculate Amazon S3 prices

I am trying to estimate how much I need to pay to store images for my customers on Amazon S3. I use their calculator but the total seems to cheap. Feels like I am missing something. Each image is ...
2
votes
1answer
426 views

Change Amazon S3 storage class on A LOT of files

We are using Amazon S3 and a great deal of our files have the STANDARD storage class. We want to change this to REDUCED_REDUNDANCY class to save the extra money. However, is has been hard to find a ...
2
votes
2answers
579 views

Best way to implement Amazon CloudFront for established website

Some of you may have a solution for this, and I need to hear it. Basically, you have a website, a lot of categories ( games, blog etc ). The idea would be to use Amazon CDN for images, flv's, ...
2
votes
4answers
1k views

Full MySQL BAckup strategy

Here is our MySQL settings : 3 MySQL servers in a "replication-ring" : they're all Master and S1 is slave to S2, which is slave to S3 which is slave to S4. Up until now we would do snapshots of the ...
2
votes
1answer
338 views

Can you have private URLs in Amazon's CloudFront?

In Amazon's S3 storage system you can have sort of private URLs using a special query string and expiring the link after a while. Is such a private system possible using Amazon's CloudFront, which is ...
2
votes
1answer
58 views

How to secure the db dump taken from daily run script

In one of my mysql slave servers I have written a daily run script, which 1) stops slave, 2) takes a db dump, 3) starts slave again, 4) encrypts it, 5) copies it to my s3-bucket. I am using aws-cli ...
2
votes
1answer
88 views

Does the AWS CLI mv command also do a copy-and-delete?

I am investigating renaming bundles in AWS S3. I would prefer to use the SDK, but it appears that the SDK only supports Copy and Delete. This because the object keys are used for partitoning the ...
2
votes
2answers
748 views

Is there a way to retrieve all contents from a versioned Amazon S3 bucket as of a particular datetime

Is it possible via the AWS CLI or via some third-party tool to easily download all contents of a versioned bucket as it was on a particular datetime? I.E. I would like to retrieve the version of each ...
2
votes
1answer
1k views

What is your general strategy to backup S3 buckets?

We are starting a project which involves storing large data quantities in S3. S3 is good at scaling and we've expect to have up to 5TB and million of files in bucket. While i may trust Amazon in ...
2
votes
3answers
472 views

What is the best way to backup mysql in s3?

mysqldump is probably not the best idea on running backups every x hours but is the one that we are currently using. The backups are around 150 Megs each so sending it to other machines could waste ...
2
votes
1answer
223 views

How can I turn an existing Windows Server 2008 EBS-backed instance into a S3-backed (instance-backed) AMI?

I understand there is no direct conversion to get from EBS to S3. But is there some series of steps that can make this happen, or is rebuilding the image from scratch the best option? Is there a way ...
2
votes
1answer
710 views

S3 to cloudfront edge location vs cloudfront to end user transfer ratio

We host videos in amazon S3 and use cloudfront to deliver them. Here is last months numbers: 4,2 TB total traffic from cloudfront to end user. 4 TB of this is in europe, the rest mainly in the US. ...
2
votes
1answer
3k views

Cannot upload files bigger than 8GB to Amazon S3 by multi-part upload due to broken pipe

I implemented S3 multi-part upload, both high level and low level version, based on the sample code from http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html and ...
2
votes
1answer
863 views

Alias multiple DNS entries to one Amazon S3 Bucket

I have a bucket on Amazon S3. Lets call it "webstatic.mydomain.com". I have a DNS alias setup for that bucket webstatic.mydomain.com CNAME -> web-static.mydomain.com.s3.amazonaws.com. This all works ...
2
votes
2answers
1k views

Understanding Amazon AWS usage data

I love Amazon AWS, but I'm tearing my hair out over the obscure usage data they give you. All they seem to have is either an extremely high level summary that just shows you your total monthly cost ...
2
votes
1answer
232 views

Why would a Image Hosting website, such as Imgur, use AWS EC2 instances over S3 buckets for uploads?

I was reading a Q&A with the creator of Imgur, and he went into detail explaining the server infrastructure that Imgur runs on. Here's a small quote from what he had to say: Most of the ...
2
votes
1answer
1k views

HTTPS access to S3 bucket through Cloudflare

I'm trying to use Cloudflare to provide https access to an s3 bucket. I've setup the CNAME entry in Cloudfront with pointing to the bucket and I've verified I can access it via http but when I try and ...
2
votes
1answer
283 views

s3 policy for quota

Is there a s3 policy rule that can only allow a user to upload so many GB of files? Basically looking to set a quota on the objects a user can upload. Here is the rule set that I have now: { ...
2
votes
1answer
3k views

How to copy all s3 buckets to another AWS account?

I'm copying all EC2 instances to another AWS account (creating the AMIs, sharing them with that account and running the instances from AMIs there). I also need to move my S3 buckets since the ...
2
votes
2answers
7k views

How to config Amazon Route53 working without www in sub-domain

edit: Amazon now supports this. http://aws.typepad.com/aws/2012/12/root-domain-website-hosting-for-amazon-s3.html I have my domain config in Route53 at Amazon AWS exec ping command in my domain ...
2
votes
1answer
1k views

Best failover strategy for e-mail servers on AWS to ensure high availability

We have our e-mail server hosted on AWS. Last week Amazon had a failure in their East Coast region which brought down our server along with many others. We now want to implement a failover strategy ...
2
votes
1answer
112 views

Performance issue in using Amazon S3 for run-time updates

I have a web application hosted in Amazon EC2 as well as in Rackspace. Users can upload files through this application and also modify them. Those files are stored in local storage in each of these ...
2
votes
1answer
990 views

Use Microsoft DPM 2010 with Azure or Amazon S3?

Does anyone know of any software that hooks up with Microsoft's DPM 2010 or even the new 2012 to ship data to cloud storage providers like Azure or S3? We utilize D2D backups today but want a more ...
2
votes
1answer
601 views

Hosting animated GIF banners on S3

I have uploaded some animated GIF banners to our S3 account and when we use the web url provided by S3 and try to view the banners in a browser, instead of display the banners in the browsers, I get ...
2
votes
3answers
354 views

Bandwidth-friendly backup strategy using tar?

We've used tar to backup and compress (gzip) selected directories on our file server with very good results until recently. Each and every one of our backups are stored on mirrored (RAID) harddrives ...
2
votes
1answer
176 views

Amazon S3 Set Expires header

Is there any way to tell Amazon S3 that all existing and future files in some bucket should have some speicific Expires header?
2
votes
1answer
471 views

Simple command line storage tools for Azure Storage (like s3cmd, sdbShell)?

I'm currently backing up/migrating Amazon S3 and SimpleDB data from AWS to my on premise systems using sdbShell and s3cmd. Works great. Now we're putting an app up on Microsoft Azure and I want to ...
2
votes
1answer
137 views

Easiest way to switch many files to RRS in Amazon S3?

I have an S3 bucket with many thousands of files copied from a linux machine. I would like to switch all files within a folder to RRS (Reduced Redundancy Storage). How can I do this without deleting ...
2
votes
2answers
267 views

Can you have public and private files on the same Cloudfront distribution?

We have files serving through the Cloudfront CDN and we need to have some files to be public and some others to be private. One option would be to have two different buckets and distributions but it ...
2
votes
1answer
720 views

Is any place to change the default ACL to Amazon S3?

I host my image website to Amazon S3, and want keep direct access for everyone. When i upload new file to my S3 bucket, i must change the files acl manual. Is any idea to change the default acl for a ...
2
votes
1answer
522 views

Amazon S3 access control + bandwidth

I'm thinking about maybe using Amazon S3, but have a few questions: does it provide access control to limit GETs? does it provide "high-speed" transfer? I'm looking to serve large files to a vast ...
2
votes
1answer
2k views

What is the best way to copy large files from my workstation to a Windows Server based EC2 instance?

I would like to know if there is a better way to transfer data from my workstation to my Windows based EC2 instance. Currently I am using remote desktop copy/paste, but that is a drag as I always ...
2
votes
1answer
485 views

Amazon S3 Redirect

Is there a way to redirect your domain which you are pointing towards S3 to your home pages? Like I am hosting my videos on Amazon S3 but if someone try to access the my s3.mywebsite.com it result in ...
2
votes
1answer
1k views

CNAME causing “Server Not Found” with AWS S3

I have a goDaddy site with a CNAME record set up to look at a specific bucket in my S3 account. It works fine and loads files without problem. I have a server with SliceHost that has a CNAME record ...
2
votes
1answer
192 views

AWS: Why is the bucket size on CloudWatch higher than when using S3 tools?

I use the awscli tools to verify the size of my buckets. First the S3 call (all files without versioning, just an ordinary upload): $ aws s3 ls s3://XXX --recursive --human-readable --summarize ...
2
votes
0answers
41 views

Most bandwidth efficient setup for video sharing [closed]

I built a video sharing platform for a popular french podcast I contribute to. Initially I used S3 and linked directly to it, bad idea. Then I used a pull CDN to better optimize my bandwidth, but s3 ...
2
votes
0answers
104 views

Solution for file data backup, versioning and replication

We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
2
votes
0answers
36 views

What does a healthy CNAME to an S3 web access bucket look like? [duplicate]

I have feed.foodschmood.com configured according to Amazon's instructions (a CNAME to one of their endpoints, which is itself a CNAME), but I'm skeptical that this is a "proper" configuration. ...
2
votes
0answers
310 views

Strict-Transport-Security on CloudFront with S3 origin?

My company's site has a static homepage for speed and cost reasons. We use S3 as the origin for CloudFront. Now, we would like to declare Strict-Transport-Security for the entire domain, but S3 seems ...
2
votes
1answer
471 views

Postgresql WAL-E - Replication falling behind + can't catch up

Does anyone know how I can have my replications process the WAL segments faster while using WAL-E? https://github.com/wal-e/wal-e I have one master that leverages WAL-E to upload to S3, and I have 6 ...
2
votes
1answer
610 views

How can I whitelist oubound-from-private-subnet traffic to S3 on the NAT instance of an EC2 VPC?

I'm researching privacy-compliant high-security deployment options for AWS VPC. I'm looking for a method of tightening outbound traffic from the private subnets at the NAT instance. For example, how ...
2
votes
1answer
721 views

How backup files with logrotate and Amazon S3?

I would like to create a new standard of two things: how long would be generated and be rotated the logs generated by my applications ? how to transfer the logs to Amazon S3, as a backup server ?. ...
2
votes
0answers
2k views

How can I move s3 objects >5gb from one bucket to another?

Objects larger than 5GB are supported with multi-part uploading. We have 5gb+ files in one bucket already that I'd like to move to another, under the same AWS account. When I issue the command using ...
2
votes
1answer
1k views

Using Amazon s3 and Cloudfront what is the best way to set expires headers and cache expiration

We have a huge bucket of photos on s3. We add and remove from this daily but it currently has millions of photos that we don't want to rewrite or update if possible. We have not set any expires ...
2
votes
1answer
731 views

Amazon S3 Website & MySQL Backup

I have a website (a digital asset management system/gallery - http://www.resourcespace.org) that has a huge amount of images. The total size of the website, including the images is approximately 6gb. ...
2
votes
0answers
233 views

Gluster with s3ql mounted brick?

I am trying to test run Gluster with a volume consisting of two local bricks, the first being a plain directory and the second being a s3ql based mount. Having just the plain directory as a unique ...
2
votes
0answers
472 views

Web-based file manager like File Thingie, but for S3?

At work, we use a little PHP script called File Thingie, which is a web-based file manager. Basically, an FTP replacement. It allows users to upload and download files, and most importantly, it has a ...
2
votes
1answer
581 views

How to access an Eucalyptus Walrus(S3) using Amazon AWS SDK

I have downloaded the Amazon AWS SDK for C#, I have no problem accessing the EC2 part of our private cloud running Eucalyptus, I can list, Images, Instances, Zones ... This is working fine : ...