Questions tagged [amazon-s3]
This is for questions on Amazon's cloud storage service.
1,134
questions
392
votes
27
answers
476k
views
How can I get the size of an Amazon S3 bucket?
I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data.
The s3cmd tools provide a way to get the total file size using s3cmd ...
159
votes
8
answers
163k
views
Amazon Cloudfront with S3. Access Denied
We're trying to distribute out S3 buckets via Cloudfront but for some reason the only response is an AccessDenied XML document like the following:
<Error>
<Code>AccessDenied</Code&...
111
votes
3
answers
66k
views
How long before an s3 bucket can be created with same name after deletion?
Title is quite explanatory, but I have just deleted an s3 bucket as it was in the wrong region and am wanting to recreate it in the correct region with same name as the just deleted one.
Is there any ...
80
votes
8
answers
203k
views
Copy directory structure intact to AWS S3 bucket
I want to use the AWS S3 cli to copy a full directory structure to an S3 bucket.
So far, everything I've tried copies the files to the bucket, but the directory structure is collapsed. (to say it ...
68
votes
2
answers
20k
views
Why does AWS recommend against public S3 buckets?
"We highly recommend that you never grant any kind of public access to your S3 bucket."
I have set a very granular public policy (s3:GetObject) for one bucket that I use to host a website. Route53 ...
64
votes
8
answers
274k
views
What is causing Access Denied when using the aws cli to download from Amazon S3?
I'm really flailing around in AWS trying to figure out what I'm missing here. I'd like to make it so that an IAM user can download files from an S3 bucket - without just making the files totally ...
55
votes
8
answers
49k
views
Chrome S3 Cloudfront: No 'Access-Control-Allow-Origin' header on initial XHR request
I have a webpage (https://smartystreets.com/contact) that uses jQuery to load some SVG files from S3 through the CloudFront CDN.
In Chrome I will open an Incognito window as well as the console. Then ...
53
votes
3
answers
82k
views
Rsync to AWS S3 bucket
For a server I am hosting a website on I want to backup the data and settings to an S3 bucket. I found out that you can't directly use rsync to backup to an S3 bucket. Is there another way to achieve ...
53
votes
6
answers
67k
views
How to move files between two S3 buckets with minimum cost?
I have millions of files in a Amazon S3 bucket and I'd like to move these files to other buckets and folders with minimum cost or no cost if possible. All buckets are in the same zone.
How could I ...
50
votes
6
answers
76k
views
How do I back up an AWS S3 Bucket without versioning the source bucket [closed]
Is there any way to recover from accidental deletions of an Amazon S3 Bucket?
We've got critical info in our buckets and I need to mitigate the risk of accidental or malicious deletions of the bucket ...
47
votes
7
answers
65k
views
How can one efficiently use S3 to back up files incrementally?
I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP.
There look to be a few approaches.
s3rsync ...
46
votes
11
answers
82k
views
Most efficient way to batch delete S3 Files
I'd like to be able to batch delete thousands or tens of thousands of files at a time on S3. Each file would be anywhere from 1MB to 50MB. Naturally, I don't want the user (or my server) to be waiting ...
42
votes
7
answers
66k
views
Using RSYNC with Amazon S3 [closed]
I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools)
Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
40
votes
2
answers
10k
views
AWS S3 charging for 4 TB of storage when only using less than 1 GB
I'm having trouble understanding my large S3 bill, and figured I'd ask here before dropping $30 on AWS monthly support.
Basically, I have an Amazon EC2 instance that makes an API to different ...
37
votes
1
answer
35k
views
using CloudFormation with an existing S3 bucket
Using CloudFormation, I want to set some of the properties in AWS::S3::Bucket on an existing bucket. In other words, I don't want to create the bucket, I just want to enforce some of the settings. ...
34
votes
2
answers
41k
views
No Cache-Control Header for files from AWS CloudFront with S3 Origin
We just migrated to Amazon AWS. We currently have an EC2 instance that's working well. It's running Nginx in front and Apache in the back-end. That's running well also. All sites are launched properly ...
33
votes
3
answers
18k
views
How can I reuse existing resources in CloudFormation?
I have an S3 bucket as a resource in my CloudFormation template, with DeletionPolicy set to Retain. This works as expected: when deleting the stack, it does indeed retain the bucket. However, when I ...
32
votes
3
answers
45k
views
What is the difference between Buckets and Folders in Amazon S3?
What is the difference between Buckets and Folders in Amazon S3 ? Is such a thing like Folder exist in Amazon S3 ? or only the S3 clients present Folders to us for better handling ?
31
votes
8
answers
88k
views
Automatically delete old items from s3 bucket
Is there an easy way to set up a bucket in s3 to automatically delete files older than x days?
29
votes
7
answers
42k
views
How to verify a AWS VPC (S3) endpoint works?
I added a VPC endpoint to my VPC using CloudFormation, and allowed s3 usage. The routes are visible in the AWS console, but not in the local routing tables of the
EC2 instances:
$ route -n
Kernel IP ...
28
votes
4
answers
76k
views
Setting Access-Control-Allow-Origin on Cloudfront
I am having problems serving static assets to Firefox using AWS Cloudfront.
Chrome works perfect, but Firefox is returning a CORS error.
If I execute curl , I get:
HTTP/1.1 200 OK
Content-Type: ...
27
votes
4
answers
32k
views
Why S3 website redirect location is not followed by CloudFront?
I have a website hosted on Amazon S3. It is the new version of an old website hosted on WordPress.
I have set up some files with the metadata Website Redirect Locationto handle old location and ...
26
votes
5
answers
20k
views
How to move S3 bucket to different location
We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to ...
25
votes
1
answer
21k
views
How to let user upload files to S3 bucket, but not overwrite or delete?
I have the following IAM policy for a user
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1395161912000",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
...
24
votes
2
answers
46k
views
Deleting S3 files with a given prefix only
We have a bucket with more than 500,000 objects in it.
I'm assigned a job where I've to delete files which have a specific prefix. There are around 300,000 files with the given prefix in the bucket.
...
22
votes
2
answers
25k
views
No targets available when trying to set alias target from Route 53 to S3
I'm trying to setup a simple Amazon AWS S3 based website, as explained here.
I've setup the S3 bucket (simples3websitetest.com), gave it the (hopefully) right permissions:
{
"Version": "2012-10-...
21
votes
8
answers
31k
views
Which is the fastest way to copy 400G of files from an ec2 elastic block store volume to s3?
I have to copy 400G of files from an elastic block store volume to an s3 bucket... Those are about 300k files of ~1Mb
I've tried s3cmd and s3fuse, both of them are really, really slow.. s3cmd ran for ...
21
votes
2
answers
16k
views
Restore a versioned S3 bucket to a particular point in time
Let's say I've got S3 versioning enabled for my bucket: http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html
Then, let's say someone (for example, junior employee) messes up the S3 bucket (...
20
votes
2
answers
27k
views
Why don't I have permissions to edit an S3 bucket policy when logged on as the person who created the AWS account
I am new to AWS. I am setting up an S3 bucket that I want to use to store media files for a Django App I am developing.
I am logged in as the person who created the AWS account, but when I click on ...
20
votes
5
answers
13k
views
Any need to make backup of data on Amazon S3?
I'm hosting 200 GB of product images at S3 (this is my primary file host).
Do I need to back that data up somewhere else, or is S3 safe as it is?
I have been experimenting with mounting the S3 ...
18
votes
3
answers
34k
views
How do I prevent deletion of S3 buckets?
It looks like creating a bucket policy might do the trick, but I am having trouble creating the policy.
17
votes
4
answers
16k
views
Recursively changing the content-type for files of a given extension on Amazon S3
I have a large S3 bucket with a nested "folder" structure containing (among other things) static .json and .md files. Theses files are being served by S3 as text/plain rather than the correct ...
16
votes
5
answers
25k
views
How to make s3fs work with IAM roles?
We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting.
But that doesn't ...
16
votes
3
answers
10k
views
Backing up data stored on Amazon S3
I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate ...
15
votes
4
answers
46k
views
Amazon S3 Console: How to find total number of files with in a folder? [duplicate]
So I can see S3 console and can get into my folder with files listing but I don't find way to find total number of files in it other than going thru pagination which does not work considering ...
15
votes
5
answers
2k
views
Software for using Amazon S3 as backup service [closed]
What software do you recommend for using Amazon S3 as a placeholder for daily backups?
I need it to be fully automated, and easy to restore data.
15
votes
3
answers
39k
views
How to config Amazon Route53 working without www in sub-domain
edit: Amazon now supports this. http://aws.typepad.com/aws/2012/12/root-domain-website-hosting-for-amazon-s3.html
I have my domain config in Route53 at Amazon AWS
exec ping command in my domain ...
14
votes
3
answers
3k
views
Access Amazon S3 Static Website Through IPv6
I host a static website on Amazon S3 with DNS hosted on Amazon Route 53. I'd like my site to be accessible via IPv4 and IPv6 (because it's about time).
When I use the Route 53 AWS Management Console ...
14
votes
5
answers
39k
views
Accessing Amazon S3 from a private VPC subnet
If I have a VPC running and some servers located in the private portion of that network that do backend processing by downloading files from amazon s3, can I access S3 internally to get at those files?...
14
votes
7
answers
17k
views
What is a practical way to mirror an Amazon S3 bucket?
I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the ...
14
votes
3
answers
25k
views
Forcing CloudFront to pass-through the latest HTML file from S3
Background
I'm hosting a static site on S3, with CloudFront over the top. The issue I have is with my HTML files.
According to CloudFront's FAQ:
Amazon CloudFront uses these cache control headers to ...
14
votes
3
answers
37k
views
Why is uploading to S3 so slow?
I am using s3cmd to upload to S3:
# s3cmd put 1gb.bin s3://my-bucket/1gb.bin
1gb.bin -> s3://my-bucket/1gb.bin [1 of 1]
366706688 of 1073741824 34% in 371s 963.22 kB/s
I am uploading ...
14
votes
3
answers
30k
views
Wildcard Subdomain (*.example.com) with Amazon Route53 DNS and S3
I am hosting a static website using Amazon Route53 for DNS and S3 for html files.
It is working fine.
Current Setup is:
example.com. A ALIAS s3-website-us-east-1.amazonaws.com.
www.example.com. ...
14
votes
10
answers
12k
views
S3-compatible Object Storage systems [closed]
Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system?
What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 ...
13
votes
2
answers
29k
views
Uploading files with similar filename in Amazon S3
Basically, how does Amazon S3 handle files which already have the same filename inside a specified bucket? Assuming I can only upload to one particular bucket, will S3 change the filename or something?...
13
votes
1
answer
6k
views
Duplicity and Amazon S3 Script
I'm looking to backup my linux server to Amazon S3 using duplicity.
I found a great resource here that helped me get it setup and am using the basic script that was listed there for me and now copied ...
13
votes
1
answer
3k
views
What does new "S3 Increased Request Rate Performance" annoucement mean
On 17 July 2018 there was an official AWS announcement explaining that there is no longer any need to randomize the first characters of every S3 object key to achieve maximum performance: https://aws....
13
votes
1
answer
10k
views
Serving website from Cloudfront and S3 without public bucket
When serving a website via CloudFront from an S3 bucket, I would typically apply the following Bucket Policy to allow access to my files:
{
"Version": "2012-10-17",
"Statement": [
{
...
12
votes
8
answers
41k
views
How to schedule automatic (daily) snapshots of AWS EC2 Windows Instance?
I have some Windows servers hosted on Amazon EC2. Some run Windows Server 2003 and other run Windows Server 2008. These are EBS-backed instances. Most of the instances also have some additional EBS-...
12
votes
1
answer
9k
views
Elastic Beanstalk S3 access via .ebextensions
I have a simple file in my .ebextensions folder:
00-myconfig.config
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Access:
...