Questions tagged [amazon-s3]

This is for questions on Amazon's cloud storage service.

Filter by
Sorted by
Tagged with
9 votes
3 answers
3k views

What's the easiest way to back up EC2 instances automatically?

I'm worried I might log in one day and my fully configured instance has been removed. I would like to be able to back it up daily to S3 but frustratingly this is not a simple setting in the console. ...
chrism2671's user avatar
  • 2,629
392 votes
27 answers
476k views

How can I get the size of an Amazon S3 bucket?

I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data. The s3cmd tools provide a way to get the total file size using s3cmd ...
powdahound's user avatar
  • 4,375
159 votes
8 answers
163k views

Amazon Cloudfront with S3. Access Denied

We're trying to distribute out S3 buckets via Cloudfront but for some reason the only response is an AccessDenied XML document like the following: <Error> <Code>AccessDenied</Code&...
Jordan Adams's user avatar
  • 1,691
55 votes
8 answers
49k views

Chrome S3 Cloudfront: No 'Access-Control-Allow-Origin' header on initial XHR request

I have a webpage (https://smartystreets.com/contact) that uses jQuery to load some SVG files from S3 through the CloudFront CDN. In Chrome I will open an Incognito window as well as the console. Then ...
SunSparc's user avatar
  • 1,038
28 votes
4 answers
76k views

Setting Access-Control-Allow-Origin on Cloudfront

I am having problems serving static assets to Firefox using AWS Cloudfront. Chrome works perfect, but Firefox is returning a CORS error. If I execute curl , I get: HTTP/1.1 200 OK Content-Type: ...
Tony's user avatar
  • 445
26 votes
5 answers
20k views

How to move S3 bucket to different location

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to ...
skrat's user avatar
  • 515
7 votes
1 answer
18k views

Url rewriting in Amazon S3

I have a webapp hosted in Amazon S3, the webapp is build with jQuery Mobile and I use changePage lo navigate between pages, I'm not using the hash operator(#) when I change the page. I'm having ...
Fernando P. G.'s user avatar
3 votes
2 answers
14k views

Cross Region S3 bucket policy

I would like to allow EC2 servers based in us-east-1 to read content from a bucket in us-west-2. Every time I try I get a "The Bucket you are attempting to access must be addressed using the ...
Martin's user avatar
  • 563
0 votes
1 answer
2k views

AWS S3 Static Hosting with Custom Domain - routing bare domain

I am trying to host some static content in an Amazon S3 bucket and to have that content available through my own domain. The bucket in question is called "e.tl-dr.org" and I have "index.html" set as ...
Alan Faubert's user avatar
0 votes
0 answers
2k views

AWS S3 bucket policy for get and upload

I have two buckets each with their own goals. So far I cannot understand the complicated configuration of bucket settings. It seems there are three things to configure Block public access Bucket ...
pbuzz007's user avatar
  • 133
68 votes
2 answers
20k views

Why does AWS recommend against public S3 buckets?

"We highly recommend that you never grant any kind of public access to your S3 bucket." I have set a very granular public policy (s3:GetObject) for one bucket that I use to host a website. Route53 ...
Andrew Johnson's user avatar
53 votes
3 answers
82k views

Rsync to AWS S3 bucket

For a server I am hosting a website on I want to backup the data and settings to an S3 bucket. I found out that you can't directly use rsync to backup to an S3 bucket. Is there another way to achieve ...
Bart Bergmans's user avatar
46 votes
11 answers
82k views

Most efficient way to batch delete S3 Files

I'd like to be able to batch delete thousands or tens of thousands of files at a time on S3. Each file would be anywhere from 1MB to 50MB. Naturally, I don't want the user (or my server) to be waiting ...
SudoKill's user avatar
  • 563
42 votes
7 answers
66k views

Using RSYNC with Amazon S3 [closed]

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools) Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
alex's user avatar
  • 1,730
34 votes
2 answers
41k views

No Cache-Control Header for files from AWS CloudFront with S3 Origin

We just migrated to Amazon AWS. We currently have an EC2 instance that's working well. It's running Nginx in front and Apache in the back-end. That's running well also. All sites are launched properly ...
jarvis's user avatar
  • 2,006
33 votes
3 answers
18k views

How can I reuse existing resources in CloudFormation?

I have an S3 bucket as a resource in my CloudFormation template, with DeletionPolicy set to Retain. This works as expected: when deleting the stack, it does indeed retain the bucket. However, when I ...
vartec's user avatar
  • 6,237
21 votes
2 answers
16k views

Restore a versioned S3 bucket to a particular point in time

Let's say I've got S3 versioning enabled for my bucket: http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Then, let's say someone (for example, junior employee) messes up the S3 bucket (...
elsurudo's user avatar
  • 313
18 votes
3 answers
34k views

How do I prevent deletion of S3 buckets?

It looks like creating a bucket policy might do the trick, but I am having trouble creating the policy.
Jason Watts's user avatar
16 votes
3 answers
10k views

Backing up data stored on Amazon S3

I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate ...
Fiver's user avatar
  • 213
16 votes
5 answers
25k views

How to make s3fs work with IAM roles?

We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting. But that doesn't ...
sriram's user avatar
  • 297
13 votes
1 answer
10k views

Serving website from Cloudfront and S3 without public bucket

When serving a website via CloudFront from an S3 bucket, I would typically apply the following Bucket Policy to allow access to my files: { "Version": "2012-10-17", "Statement": [ { ...
Steve's user avatar
  • 231
12 votes
1 answer
15k views

Adding CloudFlare to S3 bucket

I have an Amazon S3 bucket that contains thousands of JPGs and similar media assets. (It's the storage for my forums.) I don't host a static site in this bucket (no HTML of any kind) and it's ...
Eric's user avatar
  • 1,127
11 votes
1 answer
12k views

AWS CLI throws "Unable to locate credentials", the second time it's run

I'm trying to download some files from S3 to an EC2 instance using the AWS CLI (1.8.7) on startup using a User-Data Script. This instance has a IAM Role & Instance Profile with the appropriate ...
Simon Paris's user avatar
10 votes
8 answers
1k views

S3 Interfaces for Backup, etc [closed]

Since none of the S3 "file systems" are compatible right now, what s3 gateway tool do you recommend for mounting a remote S3 bucket for backups? I've been using S3FS, but I've heard a lot of people ...
Mikeage's user avatar
  • 2,741
8 votes
3 answers
14k views

Is mounting S3 buckets directly into EC2 instances safe?

as the title of my question says, I would like to know whether this is a good practice or not, I know I can mount it in an EC2 instance but I was wondering if, as I will be adding more instances to ...
Enrico Tuttobene's user avatar
8 votes
1 answer
6k views

S3 logs do not appear in targeted bucket

I have two buckets, one named A and another named logs. I went to the permissions page for A and enabled service logging, and set the target to logs bucket. According to the AWS documentation, this ...
jeebface's user avatar
  • 265
8 votes
2 answers
13k views

I'm trying to use "aws s3 sync" on my EC2 instance. Is the '--exclude' option broken?

I'm trying to backup my EC2 instance to S3 using Amazon's official tools that come preinstalled on the EC2 instance. I'm having difficulty getting the sync command to exclude directories from the sync....
user158845's user avatar
8 votes
5 answers
30k views

How can I set up an SFTP server backed by S3 (or similar)

I need to set up an SFTP server that, essentially, has very large capacity. I need to give one of our partners SFTP login details to a server where they will upload millions of files, totalling a few ...
Daniel Magliola's user avatar
6 votes
1 answer
18k views

Access denied copying files using S3 CLI

Attempting to pull down the contents of an S3 bucket using the AWS CLI, I'm getting the following: aws s3 cp --region us-east-1 s3://s3.amazonaws.com/my-bucket . --recursive A client error (...
KevinD's user avatar
  • 163
6 votes
3 answers
2k views

Amazon CloudFront and EC2: Global Load Balancing

We have an app that is going to store and serve up a decent amount of data in S3 to a global audience where latency should be minimized. So, we've been doing tests with Amazon CloudFront and have seen ...
Matt Rogish's user avatar
  • 1,512
5 votes
1 answer
4k views

What AWS service to avoid CORS issues with S3 static content and aws backends

Introduction I've got the following architecture deployed on Amazon AWS. The goal is to expose a web application (single page application) acting as an entrypoint at https://app.acmecorp.com. This ...
ddewaele's user avatar
  • 333
5 votes
2 answers
7k views

Getting files from an s3 bucket using IAM role credentials

I am trying to retrieve some files from a private s3 bucket to a filesystem location elastic beanstalk ec2 instance, but with no success. I've created a bucket named dev-config containing a file ...
diffa's user avatar
  • 141
5 votes
3 answers
4k views

Any way to enable Amazon S3 Versioning for existing objects?

All the information that I have read on the AWS S3 Object Versioning specifically mentions this: Objects stored in your bucket before you set the versioning state have a version ID of null. When ...
Valien's user avatar
  • 193
5 votes
1 answer
8k views

Using an IP Address to Point to Amazon S3 Storage

Is there a way to get an IP address to refer to Amazon S3 Storage location? The current solution my company is thinking of going with involves setting up a EC2 server with an Elastic IP and using ...
Tyndall's user avatar
  • 591
5 votes
1 answer
10k views

Setting up IIS reverse proxy to preserve host headers

I have an IIS server that is hosting a number of sites and apis. These sites include Confluence and Jira instances. These products actually run their own web servers so the Application Request Routing ...
Chris Becke's user avatar
4 votes
2 answers
2k views

Restrict access to S3 bucket folders to specific website users? (not using IAM Users)

I have a website where users need to log in. They can upload and delete their own pictures BUT these pictures are supposed to be private so images are not set to public that anyone can view. I know ...
Dora's user avatar
  • 341
4 votes
4 answers
5k views

Best remote mounted filesystem based on S3 [closed]

I need to know what the best (read: highly performant and fault tolerant) filesystem solution based on Amazon's S3 is. I have looked a s3fs, but I notice that it has not been worked on for a while now,...
QWade's user avatar
  • 927
3 votes
2 answers
12k views

Jenkins global AWS credentials not working via `withAWS` step

I have a Jenkins v2.62 instance with the pipeline plugin installed and successfully usable. I have AWS credentials that are tested and working: Now my pipeline job has something like this: node("...
scorpiodawg's user avatar
3 votes
1 answer
1k views

AWS: SES Rule: S3 Action: fails when choosing bucket that has encryption enabled

I can not create a SES rule to put emails into a S3 bucket that has encryption enabled (on the bucket). Created a bucket and enabled encryption. Add a SESPut bucket policy to allow SES. https://docs....
eugenevd's user avatar
  • 419
3 votes
1 answer
2k views

Trying to receive emails AND store them into an S3 bucket

I'm trying to store emails i receive into an s3 bucket, i followed this tutorial and multiple others : https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-getting-started.html My MX ...
logax's user avatar
  • 139
2 votes
4 answers
4k views

Duplicity backup to S3: BackendException

I'm trying to configure my Duplicity installation to backup to an Amazon S3 bucket. I keep getting a BackendException error with no further details. EDIT: I've removed my Duply configuration from ...
A.M.K's user avatar
  • 163
2 votes
1 answer
602 views

Amazon S3 access control + bandwidth

I'm thinking about maybe using Amazon S3, but have a few questions: does it provide access control to limit GETs? does it provide "high-speed" transfer? I'm looking to serve large files to a vast ...
Jonny's user avatar
  • 153
2 votes
1 answer
2k views

AWS S3 costs for when AWS EMR uses it

When I run an AWS EMR cluster and it reads from and writes to an AWS S3 bucket (or multiple buckets), what are the costs for that data transfer? Is that data transfer? Free, because it's ...
Marco's user avatar
  • 123
2 votes
1 answer
2k views

Best failover strategy for e-mail servers on AWS to ensure high availability

We have our e-mail server hosted on AWS. Last week Amazon had a failure in their East Coast region which brought down our server along with many others. We now want to implement a failover strategy ...
Stanley's user avatar
  • 365
2 votes
1 answer
869 views

AWS SES Encryption vs S3 bucket encryption

SES Configuration for a SES rule allows one to have the email be put into a S3 bucket. Under this configuration, the option "KMS Key" is available, which would have SES encrypt the email before ...
eugenevd's user avatar
  • 419
2 votes
2 answers
5k views

Synchronize AWS EC2 instances

I have been looking for a way to synchronize files between my EC2 web servers . However in most of the places it is mentioned that S3 can be used in such cases . However my question is S3 provides for ...
AnuragD's user avatar
  • 133
1 vote
3 answers
1k views

Amazon S3 tools for Debian?

I need to (programmatically, in a shell script) upload an EAR file to an Amazon S3 bucket on Debian (5.0.4). What, if any, Debian package provides simple, scriptable tools for that? Ideally this ...
Jonik's user avatar
  • 3,021
1 vote
2 answers
3k views

How to configure AWS Route 53 to get equivalent of domain forwarding with masking

(I apologize in advance for not having enough technical depth to ask this intelligently) I currently have an AngularJS static website hosted on AWS S3. I'm using AWS Route 53 for DNS. I'm able to ...
JET's user avatar
  • 113
1 vote
1 answer
382 views

Is there any way to persist files for a web application using Amazon AWS without a full VM?

I would like to migrate an web application to Amazon AWS. The application stores and retrieves small persistent data from a content/ directory. I tried Amazon Elastic Beanstalk to deploy the ...
Daniel Santos's user avatar
1 vote
3 answers
6k views

Is possible to reduce AWS data transfer costs using any other AWS resource?

I do have a AWS EC2 instance that contains a very large blog. My data transfer is about 200Gb/month. 80% of it are only media and images. My data transfer costs are high due the server location in ...
Daniel Santos's user avatar