Tag Info

New answers tagged

0

If you don't want to use the command-line, on Windows and OSX, there's a general purpose remote file management app called Cyberduck. Log into S3 with your access/secret key pair, right-click on the directory, click Calculate.


0

I know this is an older question but here is a PowerShell example: Get-S3Object -BucketName <buckename> | select key, size | foreach {$A += $_.size} $A contains the size of the bucket, and there is a keyname parameter if you just want the size of a specific folder in a bucket.


1

Using the official AWS s3 command line tools: aws s3 ls s3://bucket/folder --recursive | awk 'BEGIN {total=0}{total+=$3}END{print total/1024/1024" MB"}'


2

One idea would be to use the stdout generated by Unison while it is running. There is some junk in stdout that Unison uses to create a dynamic effect in the terminal while it is "looking for changes." This junk can be removed fairly easily by deleting every line containing a Carriage Return (CR) character (in vim this would be something like ...


0

You will have to create/modify a bucket policy or ACL to use the OAI. See near the bottom of this page for step-by-step instructions: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html


2

I believe sync is the method you want. Try this instead: aws s3 sync ./logdata s3://bucketname/


0

Depending on how many IAM users you have in the account, you can specify the accounts in both statements as follows (try one or two just as a test to see if it works). I've had issues in the past with getting arn:aws:iam::XXXXXXXXXXXX:root to actually cover all IAM accounts. You could also try just specifying the IAM users and remove the root entry. { ...


1

You can also consider seting up Cross-Region Replication, which allows you to both copy and keep in sync data between regions https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/


0

shell script to delete old buckets using s3cmd utility source : http://shout.setfive.com/2011/12/05/deleting-files-older-than-specified-time-with-s3cmd-and-bash/ #!/bin/bash # Usage: ./deleteOld "bucketname" "30 days" s3cmd ls s3://$1 | while read -r line; do createDate=`echo $line|awk {'print $1" "$2'}` createDate=`date -d"$createDate" +%s` ...


0

Following way uses AWS PHP SDK to get the total size of the bucket. // make sure that you are using correct region (where the bucket is) to get new Amazon S3 client $client = \Aws\S3\S3Client::factory(array('region' => $region)); // check if bucket exists if (!$client->doesBucketExist($bucket, $accept403 = true)) { return false; } // get bucket ...


0

Check if your original instance supported para-virtualization or HVM. This value can be found on your original EC2 instance's description section as 'Virtualization' and also on the AMI's description section as 'Virtualization type'. When making AMI using the EBS instead of the EC2 instance, this value could be set incorrectly (the values must match so ...


0

Your scenario sounds like exactly what the s3 sync tool is supposed to be used for. I would think "aws s3 sync local_directory s3://your_bucket_location" should work exactly like you're asking. Are you using the AWS CLI tools? If so, can you try with --dry-run and let us know if it thinks the difference is ~40k or it is actually all 2M+ files? EDIT: s3 ...


0

Without knowing how you're managing the s3 buckets, this may or may not be particularly useful. The AWS CLI tools has an option called "sync" which can be particularly effective to ensure s3 has the correct objects. If you, or your users, are managing S3 from a local filesystem, you may be able to save a ton of work determining which objects need to be ...


2

AWS supports bulk deletion of up to 1000 objects per request using the S3 REST API and its various wrappers. This method assumes you know the S3 object keys you want to remove (that is, it's not designed to handle something like a retention policy, files that are over a certain size, etc). The S3 REST API can specify up to 1000 files to be deleted in a ...


-1

in order to access your AMI's after you created and image of it, it is created and stored in your AMI'S you will find this when you are launching a new instance to the left side of your page , you can see options like community AMI, market place..etc. over there you must find a column "MY AMI's" and there you will find your AMI which you created.


0

The current EBS Snapshot capabilities for Windows Instances are explained here: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ebs-creating-snapshot.html In order to get a consistent snapshot, you have to pause writes to the volume long enough for the snapshot to complete. Typically, this means you that you have to unmount the volume. The AWS doc ...


1

Does your proxy accept wildcards? You'll want to whitelist *.s3.amazonaws.com at least, and most likely *.s3-*.amazonaws.com if you point at region-specific endpoints.


1

s4cmd is the fastest way I've found (a command-line utility written in Python): pip install s4cmd Now to calculate the entire bucket size using multiple threads: s4cmd du s3://bucket-name


0

AWS provides a list of their public IP ranges via JSON. (perhaps other formats as well, but I'm not certain.) Building a tool to ensure the JSON is parsed and the proper firewall rules are applied should be relatively straightforward. Please see the following blog article for further information. :) https://aws.amazon.com/blogs/aws/aws-ip-ranges-json/


1

While the first answer suggesting a blind timeout will likely work, might I suggest looking into S3 notification events. If you plan to scale out to more than two servers, using something like SNS or SQS to manage what happen once an object has been successfully written, for example, is going to make things much easier than relying on blanket timeouts. ...


2

is it possible to just allow the EC2 instance to access the S3? Is there any ip connected to the S3 bucket or can I set one? S3 uses many IPs. I suspect it would be difficult to nail down a list of them all. Additionally, there is no IP-to-bucket mapping, and it is not possible for you to specify an IP for a bucket. S3 is a managed service that AWS ...


1

Here is a handy little php script I used to restore my bucket from a back up bucket with versioning: To run back up into a bucket with versioning turned on #CONFIGURATION export bucket_to_backup='example-of-a-bucket-name'; export backup_bucket='example-of-a-backup-bucket-name'; export AWS_ACCESS_KEY_ID=ASDFDSAFADSFDASF; export ...



Top 50 recent answers are included