Questions tagged [amazon-s3]

This is for questions on Amazon's cloud storage service.

Filter by
Sorted by
Tagged with
392 votes
27 answers
476k views

How can I get the size of an Amazon S3 bucket?

I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data. The s3cmd tools provide a way to get the total file size using s3cmd ...
powdahound's user avatar
  • 4,375
159 votes
8 answers
163k views

Amazon Cloudfront with S3. Access Denied

We're trying to distribute out S3 buckets via Cloudfront but for some reason the only response is an AccessDenied XML document like the following: <Error> <Code>AccessDenied</Code&...
Jordan Adams's user avatar
  • 1,691
111 votes
3 answers
66k views

How long before an s3 bucket can be created with same name after deletion?

Title is quite explanatory, but I have just deleted an s3 bucket as it was in the wrong region and am wanting to recreate it in the correct region with same name as the just deleted one. Is there any ...
ljs.dev's user avatar
  • 1,284
80 votes
8 answers
203k views

Copy directory structure intact to AWS S3 bucket

I want to use the AWS S3 cli to copy a full directory structure to an S3 bucket. So far, everything I've tried copies the files to the bucket, but the directory structure is collapsed. (to say it ...
agentv's user avatar
  • 990
68 votes
2 answers
20k views

Why does AWS recommend against public S3 buckets?

"We highly recommend that you never grant any kind of public access to your S3 bucket." I have set a very granular public policy (s3:GetObject) for one bucket that I use to host a website. Route53 ...
Andrew Johnson's user avatar
64 votes
8 answers
274k views

What is causing Access Denied when using the aws cli to download from Amazon S3?

I'm really flailing around in AWS trying to figure out what I'm missing here. I'd like to make it so that an IAM user can download files from an S3 bucket - without just making the files totally ...
Josh Gagnon's user avatar
55 votes
8 answers
49k views

Chrome S3 Cloudfront: No 'Access-Control-Allow-Origin' header on initial XHR request

I have a webpage (https://smartystreets.com/contact) that uses jQuery to load some SVG files from S3 through the CloudFront CDN. In Chrome I will open an Incognito window as well as the console. Then ...
SunSparc's user avatar
  • 1,038
53 votes
3 answers
82k views

Rsync to AWS S3 bucket

For a server I am hosting a website on I want to backup the data and settings to an S3 bucket. I found out that you can't directly use rsync to backup to an S3 bucket. Is there another way to achieve ...
Bart Bergmans's user avatar
53 votes
6 answers
67k views

How to move files between two S3 buckets with minimum cost?

I have millions of files in a Amazon S3 bucket and I'd like to move these files to other buckets and folders with minimum cost or no cost if possible. All buckets are in the same zone. How could I ...
Daniel Cukier's user avatar
50 votes
6 answers
76k views

How do I back up an AWS S3 Bucket without versioning the source bucket [closed]

Is there any way to recover from accidental deletions of an Amazon S3 Bucket? We've got critical info in our buckets and I need to mitigate the risk of accidental or malicious deletions of the bucket ...
47 votes
7 answers
65k views

How can one efficiently use S3 to back up files incrementally?

I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP. There look to be a few approaches. s3rsync ...
Jaimie Sirovich's user avatar
46 votes
11 answers
82k views

Most efficient way to batch delete S3 Files

I'd like to be able to batch delete thousands or tens of thousands of files at a time on S3. Each file would be anywhere from 1MB to 50MB. Naturally, I don't want the user (or my server) to be waiting ...
SudoKill's user avatar
  • 563
42 votes
7 answers
66k views

Using RSYNC with Amazon S3 [closed]

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools) Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
alex's user avatar
  • 1,730
40 votes
2 answers
10k views

AWS S3 charging for 4 TB of storage when only using less than 1 GB

I'm having trouble understanding my large S3 bill, and figured I'd ask here before dropping $30 on AWS monthly support. Basically, I have an Amazon EC2 instance that makes an API to different ...
Garglesoap's user avatar
37 votes
1 answer
35k views

using CloudFormation with an existing S3 bucket

Using CloudFormation, I want to set some of the properties in AWS::S3::Bucket on an existing bucket. In other words, I don't want to create the bucket, I just want to enforce some of the settings. ...
300D7309EF17's user avatar
34 votes
2 answers
41k views

No Cache-Control Header for files from AWS CloudFront with S3 Origin

We just migrated to Amazon AWS. We currently have an EC2 instance that's working well. It's running Nginx in front and Apache in the back-end. That's running well also. All sites are launched properly ...
jarvis's user avatar
  • 2,006
33 votes
3 answers
18k views

How can I reuse existing resources in CloudFormation?

I have an S3 bucket as a resource in my CloudFormation template, with DeletionPolicy set to Retain. This works as expected: when deleting the stack, it does indeed retain the bucket. However, when I ...
vartec's user avatar
  • 6,237
32 votes
3 answers
45k views

What is the difference between Buckets and Folders in Amazon S3?

What is the difference between Buckets and Folders in Amazon S3 ? Is such a thing like Folder exist in Amazon S3 ? or only the S3 clients present Folders to us for better handling ?
user avatar
31 votes
8 answers
88k views

Automatically delete old items from s3 bucket

Is there an easy way to set up a bucket in s3 to automatically delete files older than x days?
jdp's user avatar
  • 413
29 votes
7 answers
42k views

How to verify a AWS VPC (S3) endpoint works?

I added a VPC endpoint to my VPC using CloudFormation, and allowed s3 usage. The routes are visible in the AWS console, but not in the local routing tables of the EC2 instances: $ route -n Kernel IP ...
M. Glatki's user avatar
  • 2,064
28 votes
4 answers
76k views

Setting Access-Control-Allow-Origin on Cloudfront

I am having problems serving static assets to Firefox using AWS Cloudfront. Chrome works perfect, but Firefox is returning a CORS error. If I execute curl , I get: HTTP/1.1 200 OK Content-Type: ...
Tony's user avatar
  • 445
27 votes
4 answers
32k views

Why S3 website redirect location is not followed by CloudFront?

I have a website hosted on Amazon S3. It is the new version of an old website hosted on WordPress. I have set up some files with the metadata Website Redirect Locationto handle old location and ...
Yannick Chaze's user avatar
26 votes
5 answers
20k views

How to move S3 bucket to different location

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to ...
skrat's user avatar
  • 515
25 votes
1 answer
21k views

How to let user upload files to S3 bucket, but not overwrite or delete?

I have the following IAM policy for a user { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1395161912000", "Effect": "Allow", "Action": [ "s3:ListBucket", ...
Markus Hedlund's user avatar
24 votes
2 answers
46k views

Deleting S3 files with a given prefix only

We have a bucket with more than 500,000 objects in it. I'm assigned a job where I've to delete files which have a specific prefix. There are around 300,000 files with the given prefix in the bucket. ...
Axel's user avatar
  • 481
22 votes
2 answers
25k views

No targets available when trying to set alias target from Route 53 to S3

I'm trying to setup a simple Amazon AWS S3 based website, as explained here. I've setup the S3 bucket (simples3websitetest.com), gave it the (hopefully) right permissions: { "Version": "2012-10-...
Amir Zucker's user avatar
21 votes
8 answers
31k views

Which is the fastest way to copy 400G of files from an ec2 elastic block store volume to s3?

I have to copy 400G of files from an elastic block store volume to an s3 bucket... Those are about 300k files of ~1Mb I've tried s3cmd and s3fuse, both of them are really, really slow.. s3cmd ran for ...
aseba's user avatar
  • 299
21 votes
2 answers
16k views

Restore a versioned S3 bucket to a particular point in time

Let's say I've got S3 versioning enabled for my bucket: http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Then, let's say someone (for example, junior employee) messes up the S3 bucket (...
elsurudo's user avatar
  • 313
20 votes
2 answers
27k views

Why don't I have permissions to edit an S3 bucket policy when logged on as the person who created the AWS account

I am new to AWS. I am setting up an S3 bucket that I want to use to store media files for a Django App I am developing. I am logged in as the person who created the AWS account, but when I click on ...
RLBChrisBriant's user avatar
20 votes
5 answers
13k views

Any need to make backup of data on Amazon S3?

I'm hosting 200 GB of product images at S3 (this is my primary file host). Do I need to back that data up somewhere else, or is S3 safe as it is? I have been experimenting with mounting the S3 ...
Chrille's user avatar
  • 483
18 votes
3 answers
34k views

How do I prevent deletion of S3 buckets?

It looks like creating a bucket policy might do the trick, but I am having trouble creating the policy.
Jason Watts's user avatar
17 votes
4 answers
16k views

Recursively changing the content-type for files of a given extension on Amazon S3

I have a large S3 bucket with a nested "folder" structure containing (among other things) static .json and .md files. Theses files are being served by S3 as text/plain rather than the correct ...
Gabriel Bauman's user avatar
16 votes
5 answers
25k views

How to make s3fs work with IAM roles?

We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting. But that doesn't ...
sriram's user avatar
  • 297
16 votes
3 answers
10k views

Backing up data stored on Amazon S3

I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate ...
Fiver's user avatar
  • 213
15 votes
4 answers
46k views

Amazon S3 Console: How to find total number of files with in a folder? [duplicate]

So I can see S3 console and can get into my folder with files listing but I don't find way to find total number of files in it other than going thru pagination which does not work considering ...
Volatil3's user avatar
  • 275
15 votes
5 answers
2k views

Software for using Amazon S3 as backup service [closed]

What software do you recommend for using Amazon S3 as a placeholder for daily backups? I need it to be fully automated, and easy to restore data.
Frode Lillerud's user avatar
15 votes
3 answers
39k views

How to config Amazon Route53 working without www in sub-domain

edit: Amazon now supports this. http://aws.typepad.com/aws/2012/12/root-domain-website-hosting-for-amazon-s3.html I have my domain config in Route53 at Amazon AWS exec ping command in my domain ...
romuloigor's user avatar
14 votes
3 answers
3k views

Access Amazon S3 Static Website Through IPv6

I host a static website on Amazon S3 with DNS hosted on Amazon Route 53. I'd like my site to be accessible via IPv4 and IPv6 (because it's about time). When I use the Route 53 AWS Management Console ...
user avatar
14 votes
5 answers
39k views

Accessing Amazon S3 from a private VPC subnet

If I have a VPC running and some servers located in the private portion of that network that do backend processing by downloading files from amazon s3, can I access S3 internally to get at those files?...
The Internet's user avatar
14 votes
7 answers
17k views

What is a practical way to mirror an Amazon S3 bucket?

I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the ...
Ben's user avatar
  • 173
14 votes
3 answers
25k views

Forcing CloudFront to pass-through the latest HTML file from S3

Background I'm hosting a static site on S3, with CloudFront over the top. The issue I have is with my HTML files. According to CloudFront's FAQ: Amazon CloudFront uses these cache control headers to ...
isNaN1247's user avatar
  • 1,705
14 votes
3 answers
37k views

Why is uploading to S3 so slow?

I am using s3cmd to upload to S3: # s3cmd put 1gb.bin s3://my-bucket/1gb.bin 1gb.bin -> s3://my-bucket/1gb.bin [1 of 1] 366706688 of 1073741824 34% in 371s 963.22 kB/s I am uploading ...
Tom Marthenal's user avatar
14 votes
3 answers
30k views

Wildcard Subdomain (*.example.com) with Amazon Route53 DNS and S3

I am hosting a static website using Amazon Route53 for DNS and S3 for html files. It is working fine. Current Setup is: example.com. A ALIAS s3-website-us-east-1.amazonaws.com. www.example.com. ...
Jasper's user avatar
  • 297
14 votes
10 answers
12k views

S3-compatible Object Storage systems [closed]

Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system? What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 ...
CodeMedic's user avatar
  • 304
13 votes
2 answers
29k views

Uploading files with similar filename in Amazon S3

Basically, how does Amazon S3 handle files which already have the same filename inside a specified bucket? Assuming I can only upload to one particular bucket, will S3 change the filename or something?...
Nikko's user avatar
  • 255
13 votes
1 answer
6k views

Duplicity and Amazon S3 Script

I'm looking to backup my linux server to Amazon S3 using duplicity. I found a great resource here that helped me get it setup and am using the basic script that was listed there for me and now copied ...
quickcel's user avatar
  • 438
13 votes
1 answer
3k views

What does new "S3 Increased Request Rate Performance" annoucement mean

On 17 July 2018 there was an official AWS announcement explaining that there is no longer any need to randomize the first characters of every S3 object key to achieve maximum performance: https://aws....
John Rees's user avatar
  • 233
13 votes
1 answer
10k views

Serving website from Cloudfront and S3 without public bucket

When serving a website via CloudFront from an S3 bucket, I would typically apply the following Bucket Policy to allow access to my files: { "Version": "2012-10-17", "Statement": [ { ...
Steve's user avatar
  • 231
12 votes
8 answers
41k views

How to schedule automatic (daily) snapshots of AWS EC2 Windows Instance?

I have some Windows servers hosted on Amazon EC2. Some run Windows Server 2003 and other run Windows Server 2008. These are EBS-backed instances. Most of the instances also have some additional EBS-...
Stanley's user avatar
  • 365
12 votes
1 answer
9k views

Elastic Beanstalk S3 access via .ebextensions

I have a simple file in my .ebextensions folder: 00-myconfig.config Resources: AWSEBAutoScalingGroup: Metadata: AWS::CloudFormation::Authentication: S3Access: ...
Strelok's user avatar
  • 353

1
2 3 4 5
23