446

This can now be done trivially with just the official AWS command line client: aws s3 ls --summarize --human-readable --recursive s3://bucket-name/ Official Documentation: AWS CLI Command Reference (version 2) This also accepts path prefixes if you don't want to count the entire bucket: aws s3 ls --summarize --human-readable --recursive s3://bucket-name/...


204

The AWS CLI now supports the --query parameter which takes a JMESPath expressions. This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]). This can be be run using the official AWS CLI as below and was introduced in Feb 2014 aws s3api list-objects --bucket BUCKETNAME --output json --...


170

AWS Console: As of 28th of July 2015 you can get this information via CloudWatch. If you want a GUI, go to the CloudWatch console: (Choose Region > ) Metrics > S3 AWS CLI Command: This is much quicker than some of the other commands posted here, as it does not query the size of each file individually to calculate the sum. aws cloudwatch get-metric-...


141

If you're accessing the root of your CloudFront distribution, you need to set a default root object: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DefaultRootObject.html To specify a default root object using the CloudFront console: Sign in to the AWS Management Console and open the Amazon CloudFront console at https://console.aws....


89

You're making two requests for the same object, one from HTML, one from XHR. The second one fails, because Chrome uses the cached response from the first request, which has no Access-Control-Allow-Origin response header. Why? Chromium bug 409090 Cross-origin request from cache failing after regular request is cached describes this problem, and it's a "...


85

The S3 docs used to say: When you delete a bucket, there may be a delay of up to one hour before the bucket name is available for reuse in a new region or by a new bucket owner. If you re-create the bucket in the same region or with the same bucket owner, there is no delay. But now they just say: ... it might take some time before the name can be ...


81

I've just had the same issue and while Kousha's answer does solve the problem for index.html in the root path, my problem was also with sub-directories as I used those combined with index.html to get "pretty urls" (example.com/something/ rather than "ugly" example.com/something.html) Partially it's Amazon's fault as well, because when you set up CloudFront ...


80

Yes, if you know what you're doing (edit: and everyone else with access to it does, too...), you can ignore this warning. It exists because even large organizations who should know better have accidentally placed private data into public buckets. Amazon will also send you heads-up emails if you leave buckets public in addition to the in-console warnings. ...


69

Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning. You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file: If you've ...


65

I believe sync is the method you want. Try this instead: aws s3 sync ./logdata s3://bucketname/


42

Since this question was last answered, there is a new AWS command line tool, aws. It can sync, rsync-like, between local storage and s3. Example usage: aws s3 sync s3://mybucket /some/local/dir/ If your system's python environment is set up properly, you can install AWS client using pip: pip install awscli


42

I was struggling with this, too, but I found an answer over here https://stackoverflow.com/a/17162973/1750869 that helped resolve this issue for me. Reposting answer below. You don't have to open permissions to everyone. Use the below Bucket policies on source and destination for copying from a bucket in one account to another using an IAM user Bucket to ...


42

To communicate to s3 you need to have 2 things IAM user credentials who has read-write access to s3 bucket. A client like aws-cli for bash, boto library for Python etc. once you have both, you can transfer any file from your machine to s3 and from s3 to your machine. Below is the example for aws-cli. To sync all files in a folder aws s3 sync source_folder ...


41

The privacy issue featured in ceejayoz's answer is not the only problem. Reading objects from an S3 bucket has a price. You will be billed by AWS for each download from this bucket. And if you have a lot of traffic (or if someone who wants to hurt your business starts to heavily download files all day long) it will quickly become expensive. If you want ...


40

I've hit a few pages in Google where you can set the Header in S3 for individual objects. That's really not a productive way to do it specially since in my case we are talking of several objects. Well, "productive" or not, that is how it actually is designed to work. CloudFront does not add Cache-Control: headers. CloudFront passes-through (and also ...


36

The A-record alias you create has to be the same as the name of the bucket, because virtual hosting of buckets in S3 requires that the Host: header sent by the browser match the bucket name. There's not really another practical way in which virtual hosting of buckets could be accomplished... the bucket has to be identified by some mechanism, and that ...


31

Here is an example how to do this with the aws cli tool. The cp tool allows the use of a recursive options, which I don't think the s3api tool can do. In this case, I'm fixing a bunch of SVGs. Remove the --dryrun options when you are ready to unleash it. aws s3 cp \ --exclude "*" \ --include "*.svg" \ --content-type="image/svg+xml" \ ...


31

S3 bucket names are globally unique. This means that if someone else has a bucket of a certain name, you cannot have a bucket with that same name. So if you are trying to create a bucket, and AWS says it already exists, then it already exists, either in your AWS account or someone else's AWS account.


30

The excruciatingly slow option is s3 rm --recursive if you actually like waiting. Running parallel s3 rm --recursive with differing --include patterns is slightly faster but a lot of time is still spent waiting, as each process individually fetches the entire key list in order to locally perform the --include pattern matching. Enter bulk deletion. I found ...


29

With a username like "The Internet", I'd expect you to know this. But since you asked... :) VPC's are truly private. Only traffic that you explicitly allow can transit the borders of the VPC. So, inside a VPC, instances needing access to external resources either need to be assigned an EIP (in which case they can access external resources using AWS's ...


26

You can try rclone. Setup (docs): rclone config Sync (docs): rclone sync /my/local/folder s3service:bucket-name


26

In the past S3 supported FQDN bucket names - i.e. exactly what you needed. Where FQDN = Fully Qualified Domain Name, i.e. full host name like static.mypage.com. The problem is that this only works with HTTP and not with HTTPS because there is no way to make S3 use a SSL certificate with your bucket name / host name (static.mypage.com). You can still do it if ...


25

When crafting Amazon IAM policies for Amazon S3, you need to be aware of the difference between Operations on the Service (e.g. ListAllMyBuckets), Operations on Buckets (e.g. ListBucket) and Operations on Objects (e.g. GetObject). In particular, the Resource specification of your policy needs to address the appropriate target entities according to the ...


22

Using the official AWS s3 command line tools: aws s3 ls s3://bucket/folder --recursive | awk 'BEGIN {total=0}{total+=$3}END{print total/1024/1024" MB"}' This is a better command, just add the following 3 parameters --summarize --human-readable --recursive after aws s3 ls. --summarize is not required though gives a nice touch on the total size. aws s3 ls ...


22

The new official AWS CLI natively supports most of the functionality of s3cmd. http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html aws s3 sync s3://oldbucket s3://newbucket --source-region us-west-1 --region us-west-2


21

First thing, you need to make sure that you whitelist origin header: If you want CloudFront to respect cross-origin resource sharing settings, configure CloudFront to forward the Origin header to your origin. http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-cors Also see: http:...


20

If you want to get the size from AWS Console: Go to S3 and select the bucket Click on "Management" tab Select Metrics tab By default you should see Storage Metric of the bucket


20

This is possible with s3cmd 1.5+ (link): $ mysqldump ... | s3cmd put - s3://bucket/file-name.sql


19

Good news! AWS has support for IPv6 in CloudFront and S3. AWS currently (2016-04-01) has very limited IPv6 support, only ELBs in EC2 Classic can do IPv6 – and they are being phased out in favour of VPCs. There is no support for IPv6 in Route53, S3, CloudFront, EC2 nodes or VPC-based load balancers (ELBs). Many are waiting for AWS to add IPv6 support, ...


19

This will list your objects, and in the end you'll see Total objects count, and size: aws s3 ls s3://bucketName/path/ --recursive --summarize just change bucketName with your bucket name and path is actually a folder within a bucket, if you need that as well (or remove it if you want the whole bucket) you can also use s3api from cli: aws s3api list-...


Only top voted, non community-wiki answers of a minimum length are eligible