Questions tagged [amazon-s3]
This is for questions on Amazon's cloud storage service.
61
questions
9
votes
3
answers
3k
views
What's the easiest way to back up EC2 instances automatically?
I'm worried I might log in one day and my fully configured instance has been removed. I would like to be able to back it up daily to S3 but frustratingly this is not a simple setting in the console.
...
392
votes
27
answers
476k
views
How can I get the size of an Amazon S3 bucket?
I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data.
The s3cmd tools provide a way to get the total file size using s3cmd ...
159
votes
8
answers
163k
views
Amazon Cloudfront with S3. Access Denied
We're trying to distribute out S3 buckets via Cloudfront but for some reason the only response is an AccessDenied XML document like the following:
<Error>
<Code>AccessDenied</Code&...
55
votes
8
answers
49k
views
Chrome S3 Cloudfront: No 'Access-Control-Allow-Origin' header on initial XHR request
I have a webpage (https://smartystreets.com/contact) that uses jQuery to load some SVG files from S3 through the CloudFront CDN.
In Chrome I will open an Incognito window as well as the console. Then ...
28
votes
4
answers
76k
views
Setting Access-Control-Allow-Origin on Cloudfront
I am having problems serving static assets to Firefox using AWS Cloudfront.
Chrome works perfect, but Firefox is returning a CORS error.
If I execute curl , I get:
HTTP/1.1 200 OK
Content-Type: ...
26
votes
5
answers
20k
views
How to move S3 bucket to different location
We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to ...
7
votes
1
answer
18k
views
Url rewriting in Amazon S3
I have a webapp hosted in Amazon S3, the webapp is build with jQuery Mobile and I use changePage lo navigate between pages, I'm not using the hash operator(#) when I change the page.
I'm having ...
3
votes
2
answers
14k
views
Cross Region S3 bucket policy
I would like to allow EC2 servers based in us-east-1 to read content from a bucket in us-west-2.
Every time I try I get a "The Bucket you are attempting to access must be addressed using the ...
0
votes
1
answer
2k
views
AWS S3 Static Hosting with Custom Domain - routing bare domain
I am trying to host some static content in an Amazon S3 bucket and to have that content available through my own domain.
The bucket in question is called "e.tl-dr.org" and I have "index.html" set as ...
0
votes
0
answers
2k
views
AWS S3 bucket policy for get and upload
I have two buckets each with their own goals. So far I cannot understand the complicated configuration of bucket settings. It seems there are three things to configure
Block public access
Bucket ...
68
votes
2
answers
20k
views
Why does AWS recommend against public S3 buckets?
"We highly recommend that you never grant any kind of public access to your S3 bucket."
I have set a very granular public policy (s3:GetObject) for one bucket that I use to host a website. Route53 ...
53
votes
3
answers
82k
views
Rsync to AWS S3 bucket
For a server I am hosting a website on I want to backup the data and settings to an S3 bucket. I found out that you can't directly use rsync to backup to an S3 bucket. Is there another way to achieve ...
46
votes
11
answers
82k
views
Most efficient way to batch delete S3 Files
I'd like to be able to batch delete thousands or tens of thousands of files at a time on S3. Each file would be anywhere from 1MB to 50MB. Naturally, I don't want the user (or my server) to be waiting ...
42
votes
7
answers
66k
views
Using RSYNC with Amazon S3 [closed]
I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools)
Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
34
votes
2
answers
41k
views
No Cache-Control Header for files from AWS CloudFront with S3 Origin
We just migrated to Amazon AWS. We currently have an EC2 instance that's working well. It's running Nginx in front and Apache in the back-end. That's running well also. All sites are launched properly ...
33
votes
3
answers
18k
views
How can I reuse existing resources in CloudFormation?
I have an S3 bucket as a resource in my CloudFormation template, with DeletionPolicy set to Retain. This works as expected: when deleting the stack, it does indeed retain the bucket. However, when I ...
21
votes
2
answers
16k
views
Restore a versioned S3 bucket to a particular point in time
Let's say I've got S3 versioning enabled for my bucket: http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html
Then, let's say someone (for example, junior employee) messes up the S3 bucket (...
18
votes
3
answers
34k
views
How do I prevent deletion of S3 buckets?
It looks like creating a bucket policy might do the trick, but I am having trouble creating the policy.
16
votes
3
answers
10k
views
Backing up data stored on Amazon S3
I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate ...
16
votes
5
answers
25k
views
How to make s3fs work with IAM roles?
We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting.
But that doesn't ...
13
votes
1
answer
10k
views
Serving website from Cloudfront and S3 without public bucket
When serving a website via CloudFront from an S3 bucket, I would typically apply the following Bucket Policy to allow access to my files:
{
"Version": "2012-10-17",
"Statement": [
{
...
12
votes
1
answer
15k
views
Adding CloudFlare to S3 bucket
I have an Amazon S3 bucket that contains thousands of JPGs and similar media assets. (It's the storage for my forums.) I don't host a static site in this bucket (no HTML of any kind) and it's ...
11
votes
1
answer
12k
views
AWS CLI throws "Unable to locate credentials", the second time it's run
I'm trying to download some files from S3 to an EC2 instance using the AWS CLI (1.8.7) on startup using a User-Data Script. This instance has a IAM Role & Instance Profile with the appropriate ...
10
votes
8
answers
1k
views
S3 Interfaces for Backup, etc [closed]
Since none of the S3 "file systems" are compatible right now, what s3 gateway tool do you recommend for mounting a remote S3 bucket for backups? I've been using S3FS, but I've heard a lot of people ...
8
votes
3
answers
14k
views
Is mounting S3 buckets directly into EC2 instances safe?
as the title of my question says, I would like to know whether this is a good practice or not,
I know I can mount it in an EC2 instance but I was wondering if, as I will be adding more instances to ...
8
votes
1
answer
6k
views
S3 logs do not appear in targeted bucket
I have two buckets, one named A and another named logs. I went to the permissions page for A and enabled service logging, and set the target to logs bucket. According to the AWS documentation, this ...
8
votes
2
answers
13k
views
I'm trying to use "aws s3 sync" on my EC2 instance. Is the '--exclude' option broken?
I'm trying to backup my EC2 instance to S3 using Amazon's official tools that come preinstalled on the EC2 instance. I'm having difficulty getting the sync command to exclude directories from the sync....
8
votes
5
answers
30k
views
How can I set up an SFTP server backed by S3 (or similar)
I need to set up an SFTP server that, essentially, has very large capacity. I need to give one of our partners SFTP login details to a server where they will upload millions of files, totalling a few ...
6
votes
1
answer
18k
views
Access denied copying files using S3 CLI
Attempting to pull down the contents of an S3 bucket using the AWS CLI, I'm getting the following:
aws s3 cp --region us-east-1 s3://s3.amazonaws.com/my-bucket . --recursive
A client error (...
6
votes
3
answers
2k
views
Amazon CloudFront and EC2: Global Load Balancing
We have an app that is going to store and serve up a decent amount of data in S3 to a global audience where latency should be minimized. So, we've been doing tests with Amazon CloudFront and have seen ...
5
votes
1
answer
4k
views
What AWS service to avoid CORS issues with S3 static content and aws backends
Introduction
I've got the following architecture deployed on Amazon AWS.
The goal is to expose a web application (single page application) acting as an entrypoint at https://app.acmecorp.com. This ...
5
votes
2
answers
7k
views
Getting files from an s3 bucket using IAM role credentials
I am trying to retrieve some files from a private s3 bucket to a filesystem location elastic beanstalk ec2 instance, but with no success.
I've created a bucket named dev-config containing a file ...
5
votes
3
answers
4k
views
Any way to enable Amazon S3 Versioning for existing objects?
All the information that I have read on the AWS S3 Object Versioning specifically mentions this:
Objects stored in your bucket before you set the versioning state have a version ID of null. When ...
5
votes
1
answer
8k
views
Using an IP Address to Point to Amazon S3 Storage
Is there a way to get an IP address to refer to Amazon S3 Storage location? The current solution my company is thinking of going with involves setting up a EC2 server with an Elastic IP and using ...
5
votes
1
answer
10k
views
Setting up IIS reverse proxy to preserve host headers
I have an IIS server that is hosting a number of sites and apis. These sites include Confluence and Jira instances. These products actually run their own web servers so the Application Request Routing ...
4
votes
2
answers
2k
views
Restrict access to S3 bucket folders to specific website users? (not using IAM Users)
I have a website where users need to log in. They can upload and delete their own pictures BUT these pictures are supposed to be private so images are not set to public that anyone can view.
I know ...
4
votes
4
answers
5k
views
Best remote mounted filesystem based on S3 [closed]
I need to know what the best (read: highly performant and fault tolerant) filesystem solution based on Amazon's S3 is. I have looked a s3fs, but I notice that it has not been worked on for a while now,...
3
votes
2
answers
12k
views
Jenkins global AWS credentials not working via `withAWS` step
I have a Jenkins v2.62 instance with the pipeline plugin installed and successfully usable. I have AWS credentials that are tested and working:
Now my pipeline job has something like this:
node("...
3
votes
1
answer
1k
views
AWS: SES Rule: S3 Action: fails when choosing bucket that has encryption enabled
I can not create a SES rule to put emails into a S3 bucket that has encryption enabled (on the bucket).
Created a bucket and enabled encryption.
Add a SESPut bucket policy to allow SES. https://docs....
3
votes
1
answer
2k
views
Trying to receive emails AND store them into an S3 bucket
I'm trying to store emails i receive into an s3 bucket, i followed this tutorial and multiple others :
https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-getting-started.html
My MX ...
2
votes
4
answers
4k
views
Duplicity backup to S3: BackendException
I'm trying to configure my Duplicity installation to backup to an Amazon S3 bucket. I keep getting a BackendException error with no further details.
EDIT: I've removed my Duply configuration from ...
2
votes
1
answer
602
views
Amazon S3 access control + bandwidth
I'm thinking about maybe using Amazon S3, but have a few questions:
does it provide access control to limit GETs?
does it provide "high-speed" transfer?
I'm looking to serve large files to a vast ...
2
votes
1
answer
2k
views
AWS S3 costs for when AWS EMR uses it
When I run an AWS EMR cluster and it reads from and writes to an AWS S3 bucket (or multiple buckets), what are the costs for that data transfer?
Is that data transfer?
Free, because it's ...
2
votes
1
answer
2k
views
Best failover strategy for e-mail servers on AWS to ensure high availability
We have our e-mail server hosted on AWS. Last week Amazon had a failure in their East Coast region which brought down our server along with many others.
We now want to implement a failover strategy ...
2
votes
1
answer
869
views
AWS SES Encryption vs S3 bucket encryption
SES
Configuration for a SES rule allows one to have the email be put into a S3 bucket.
Under this configuration, the option "KMS Key" is available, which would have SES encrypt the email before ...
2
votes
2
answers
5k
views
Synchronize AWS EC2 instances
I have been looking for a way to synchronize files between my EC2 web servers .
However in most of the places it is mentioned that S3 can be used in such cases . However my question is S3 provides for ...
1
vote
3
answers
1k
views
Amazon S3 tools for Debian?
I need to (programmatically, in a shell script) upload an EAR file to an Amazon S3 bucket on Debian (5.0.4). What, if any, Debian package provides simple, scriptable tools for that?
Ideally this ...
1
vote
2
answers
3k
views
How to configure AWS Route 53 to get equivalent of domain forwarding with masking
(I apologize in advance for not having enough technical depth to ask this intelligently)
I currently have an AngularJS static website hosted on AWS S3. I'm using AWS Route 53 for DNS. I'm able to ...
1
vote
1
answer
382
views
Is there any way to persist files for a web application using Amazon AWS without a full VM?
I would like to migrate an web application to Amazon AWS.
The application stores and retrieves small persistent data from a content/ directory. I tried Amazon Elastic Beanstalk to deploy the ...
1
vote
3
answers
6k
views
Is possible to reduce AWS data transfer costs using any other AWS resource?
I do have a AWS EC2 instance that contains a very large blog. My data transfer is about 200Gb/month. 80% of it are only media and images.
My data transfer costs are high due the server location in ...