This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

0
votes
2answers
899 views

Enable Cross Origin Scripting on Nginx from retina.js to load files from another URL?

Im running a retina image plugin along with W3 Cache for CDN support. W3 cache moves images to S3, but the JS file won't server the files from there due to the difference in URL (it goes from h ...
3
votes
1answer
3k views

s3put fails with “Connection reset by peer” for large files

I can upload small files quite fine using the same commands, however whenever I try to upload a 5 GB file to S3 then the s3 script fails: [backup]$ s3put --bucket john-murdoch-bucket --reduced ...
12
votes
4answers
9k views

Which is the fastest way to copy 400G of files from an ec2 elastic block store volume to s3?

I have to copy 400G of files from an elastic block store volume to an s3 bucket... Those are about 300k files of ~1Mb I've tried s3cmd and s3fuse, both of them are really, really slow.. s3cmd ran for ...
2
votes
0answers
56 views

Set up a different index document per directory with S3 static website hosting

I am using S3 static website hosting, and trying to get the following redirects to work: example.com --> example.com/index.html example.com/foo --> example.com/foo/foo.html example.com/bar ...
-2
votes
1answer
9 views

Sync files between AWS S3 and Rackspace cloudfiles

I am looking for a solution for syncing files between AWS S3 and Rackspace Cloudfiles on a regular basis. Preferably, using a third party service or low maintenance self-hosted solution.
0
votes
0answers
15 views

How to securely allow web apps to read/write to S3? (the new way with IAM)

I've been banging my head against a wall on this for two hours. It's frustrating that Amazon has made this so complicated when it doesn't need to be. Let's say I have a bucket called "test," and a ...
0
votes
1answer
20 views

Using Duplicity with AWS S3 and IAM Folder policy

I'm trying to setup backups from multiple hosts to a single bucket. Each host has their own AWS login and credentials. The top most folder (prefix) maps to the AWS user id and I'm trying to limit ...
6
votes
6answers
14k views

How to schedule automatic (daily) snapshots of AWS EC2 Windows Instance?

I have some Windows servers hosted on Amazon EC2. Some run Windows Server 2003 and other run Windows Server 2008. These are EBS-backed instances. Most of the instances also have some additional ...
1
vote
2answers
1k views

Anonymous access to S3 bucket only from my EC2 instances

Is there a way to let anonymous access to a certain S3 bucket only from my EC2 instances (all of them) within a single AWS account? I know I can use IAM roles, but I found it's just too many moving ...
0
votes
0answers
32 views

DNS Based Load balancing on squid - odd issues

I'm attempting to do round-robin DNS based load balancing for two squid boxes running as simple HTTP(S) proxies. For testing I'm using AWS Route 53. To simulate round-robin I created two records ...
0
votes
1answer
37 views

How best way to rsync from aws s3 to gcloud?

I must execute a rsync ( not only copy or move ) from all my buckets from s3 to google cloud. I need create the same infraestructure of s3 to google. I have like 2Tera in s3, and all days I receive ...
0
votes
0answers
12 views

Duplicity for remote backup to AWS S3

I am using Bacula with S3 for backups. I keep bacula volumes for 60 days or so. Using duplicity as I would like to use GPG encryption of the volumes. Is there any way to do an rsync style sync using ...
-2
votes
1answer
42 views

Use Apt Inside AWS VPC Without NAT Instance

I have several Ubuntu machines inside VPCs. For many of them, the only network access they need whatsoever is for "apt-get update && apt-get upgrade" and NTP. For NTP, I can simply have an NTP ...
9
votes
3answers
2k views

What's the easiest way to back up EC2 instances automatically?

I'm worried I might log in one day and my fully configured instance has been removed. I would like to be able to back it up daily to S3 but frustratingly this is not a simple setting in the console. ...
0
votes
1answer
976 views

Amazon Cloudfront with S3 Redirect

I'm trying to do something that should be relatively straight-forward - I want to setup a few domains and subdomains to redirect to the core domain for our site, but I want to put the redirects out on ...
0
votes
0answers
34 views

AWS security group egress rules for S3

I manually created a new security group using the AWS CLI. I created ingress rules that allow incoming connections only from my company's public IP address using the known ports for SSH (22) and ...
0
votes
0answers
16 views

Duplicity GPG key not available

I am using the following tutorial to get Duplicity to back up my mysql dumps to Amazon S3. http://kappataumu.com/articles/cloud-backups-duplicity-s3.html However I am getting an issue with the ...
0
votes
1answer
45 views

Is S3 a proper choice to keep live linux user home directories?

We are building an RDP cluster using Amazon EC2/S3. Users' home directories are on S3 (single bucket) and all EC2 nodes mount the same S3 bucket using S3FS. Each EC2 node uses the appropriate home ...
0
votes
1answer
23 views

Can I use an IAM role to grant my Heroku app access to my Amazon S3 bucket?

Heroku: Using AWS S3 to Store Static Assets and File Uploads suggests using my AWS security credentials to enable my Heroku app to access my Amazon S3 bucket. However, isn't it better practice (as ...
1
vote
2answers
1k views

s3fs Input/output error

I'm trying to set up a backup system with s3fs and the Amazon S3 service. I followed this this guide and this guide. Tailing /var/log/messages I get: Aug 28 13:37:46 server s3fs:###response=403 I ...
0
votes
1answer
28 views

AWS/Route53/S3: Redirecting www.example.com to example.com works. Redirecting example.com to www.example.com doesn't

To set up a static site on S3 with DNS on Route53, we have two options. If we want the users to type www.example.com and have that auto-replaced in the address bar by example.com, we would do the ...
0
votes
1answer
26 views

How to stop s3 from listing some of the content?

It is an example. What setting of S3 causes the s3 server to a listing of bucket content? EDIT: I have followed Céline Aussourd's answer and turned off the list of the s3 bucket. But ...
0
votes
0answers
514 views

s3cmd fails to put files

I'm trying to put mysql dump file on s3 bucket. It gives an error like ; sudo ./backup.sh WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. ...
0
votes
1answer
50 views

Amazon Cloud Formation: Import file from S3 bucket

I am struggling to get files from my S3 bucket to my EC2 instance using cloud formation. I have the following files: "files" : { "/var/www/text.txt": { "source": ...
0
votes
0answers
16 views

Uploaded 0 object(s) to bucket from Write-S3Object command

When I upload a file with the Write-S3Object command and it seems work properly and in another tool I can see the file but I get the follow message: PS> Write-S3Object -BucketName my-bucket ...
1
vote
1answer
984 views

Using TntDrive to mount Amazon S3 on a Windows Amazon EC2 instance

I currently have 2 Amazon EC2 Windows Server 2008 instances setup with a load balancer. My problem now is that I need both instances to be able to access a single file system source (I'm trying to use ...
1
vote
0answers
12 views

S3 + Cloudflare vs S3 + 2 Web Server + Cloudflate

Currently, I run two websites and each site own it own big server. Currently, all of those two sites has roughly 300GB of images. Its grow rate is 15GB of images per month. I know that eventually, one ...
0
votes
0answers
30 views

EC2 websites & RDS databases backup

We have some Amazon EC2 machines on which different websites are working, and their database is being pulled from RDS-MySQL, I know for machine I can have images as backup and for RDP-MySQL it has ...
1
vote
4answers
568 views

Good simple VPS to S3 Backup scripts?

anyone has a nice, simple script for backing up a VPS to a S3 bucket? can be ruby, python, whatever... should be easy to configure! thanks
0
votes
0answers
115 views

Nginx as s3 proxy with private buckets

I'm trying to set up Nginx as a proxy for a static html/js webapp hosted on S3. I read a lots of tutorials and posts and I could make it work if my bucket is set as public. The reason I need Nginx as ...
0
votes
0answers
59 views

duplicity s3 backup from centos

I'm trying to backup server data to S3 but getting no success. Below is the script which I run to start backup on S3. I have create a user using IAM and have granted all permissions to the user. ...
0
votes
1answer
45 views

Locating an AWS S3 physically in a private subnet (like RDS)

Similar to how an RDS instance requires you to pick the specific subnets (which can be in a VPC's private subnet), can AWS S3 stores also be physically located in a VPC's private subnet?
0
votes
1answer
16 views

What is the easiest way to duplicate an existing s3 bucket (excluding data)? [closed]

I want to be able to create a new bucket which mirror the settings of an existing bucket. What is the best way to do it? Ideally I want to be able to capture the settings via aws cli and create a ...
2
votes
4answers
5k views

Accessing Amazon S3 from a private VPC subnet

If I have a VPC running and some servers located in the private portion of that network that do backend processing by downloading files from amazon s3, can I access S3 internally to get at those ...
0
votes
0answers
12 views

How can I improve the space usage of duplicate content in S3?

In my project I have duplicated files in various folders in a S3 bucket. Based on the output of s3cmd du, each time I make a copy of some content in S3, the usage increases. I wonder does s3 support ...
3
votes
2answers
98 views

Replicated shared filesystem

I'm looking into setting up a shared filesystem/file server on AWS (EC2) infrastructure that offers replication and fairly painless failover. This filesystem would host potentially millions of files ...
0
votes
0answers
8 views

Read data from external bucket on Amazon EMR

I need to read data from a bucket that I don't own, and I get the IAM account key and secrete access key to the bucket. What is the best way I can access the data? How can I add another IAM with ...
0
votes
1answer
69 views

EC2 to S3 upload speed decrease with increase in instances

I have a job that gets data from a database, runs some code, and uploads the result to an S3 bucket. The code takes approximately 1 minute to run, and the result file is approximately 10MB. Both the ...
0
votes
3answers
752 views

how to pipe a mysql dump to s3cmd

I want to transfer a mysql dump, compressed, to s3. I tried: mysqldump -u root -ppassword -all-databases | gzip -9 | s3cmd put s3://bucket/sql/databases.sql.gz but then I get: ERROR: Not enough ...
1
vote
0answers
58 views

AWS S3 bucket permissions

I've created AWS s3 buckets with several thousand objects, initially the access policy was restricted to authenticated users, but later due to some requirement we need to make all objects in this ...
0
votes
1answer
153 views

AWSCLI s3 sync doesn't create directory

Hi try to sync a local directory to a S3 bucket. The files in the directory get synced perfectly, just the directory isn't created at all. Can you guys give me a hint how to do it? "newdir" contains ...
75
votes
17answers
52k views

How can I get the size of an Amazon S3 bucket?

I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data. The s3cmd tools provide a way to get the total file size using s3cmd ...
1
vote
0answers
58 views

jboss cluster in ec2 with s3ping

Am trying to get s3ping discovery method working in wildfly 8.0.0. I am using default Amazon Linux AMI and using chef to launch instances in ec2 based on this tutorial ...
1
vote
1answer
308 views

Getting a simple list of changed files from unison

I have a filesystem that is changed on two servers and also needs to be replicated to Amazon S3. Until recently, syncing the filesystem between the two servers using Unison, and then copying to S3 ...
0
votes
1answer
31 views

Origin access identity and S3 permission

From http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PrivateContent.html , I have already created an OAI, as shown in this screenshot Am I supposed to grant the permission to ...
0
votes
1answer
127 views

Copy directory structure intact to AWS S3 bucket

I want to use the AWS S3 cli to copy a full directory structure to an S3 bucket. So far, everything I've tried copies the files to the bucket, but the directory structure is collapsed. (to say it ...
3
votes
1answer
104 views

Deny anonymous downloads in AWS S3

I have a bucket with several files publicly downloadable. I want to put up a bucket policy after which these files should only be downloadable by my IAM users. The policy I have got so far is this: { ...
8
votes
5answers
4k views

How to move S3 bucket to different location

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to ...
4
votes
5answers
3k views

Automatically delete old items from s3 bucket

Is there an easy way to set up a bucket in s3 to automatically delete files older than x days?
1
vote
2answers
3k views

AWS - Status check fails when loading AMI created from snapshot

I have a running micro instance using an 8GB EBS that I've customized. To my understand there are two ways I can create an AMI from this. 1) EC2 console -> INSTANCES - Instances -> Right Click ...