Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data.

The s3cmd tools provide a way to get the total file size using s3cmd du s3://bucket_name, but I'm worried about its ability to scale since it looks like it fetches data about every file and calculates its own sum. Since Amazon charges users in GB-Months it seems odd that they don't expose this value directly.

Although Amazon's REST API returns the number of items in a bucket, [s3cmd] doesn't seem to expose it. I could do s3cmd ls -r s3://bucket_name | wc -l but that seems like a hack.

The Ruby AWS::S3 library looked promising, but only provides the # of bucket items, not the total bucket size.

Is anyone aware of any other command line tools or libraries (prefer Perl, PHP, Python, or Ruby) which provide ways of getting this data?

share|improve this question
add comment

9 Answers 9

up vote 53 down vote accepted

Since it's the first result for google, I reply later. s3cmd can do this : s3cmd du s3://bucket-name

share|improve this answer
    
Thanks. Here's some timing. On a bucket that holds an s3ql deduplicated filesystem with about a million files using about 33 GB of undupicated data, and about 93000 s3 objects, s3cmd du took about 4 minutes to compute the answer. I'm curious to know how that compares with other approaches like the php one described elsewhere here. –  nealmcb Jul 10 '12 at 23:46
    
It is slow because the S3 ListObjects API call returns objects in pages of 1000 objects. As I/O is by far the limiting factor I think any solution will be relatively slow over 93000 objects. –  David Caunt Apr 20 '13 at 13:54
    
s4cmd can also do the same thing, with the added benefit of multi-threading the requests to S3's API to compute the result faster. The tool hasn't been updated recently, but the Internet passer-by may find it useful. –  Nick Chammas Jul 7 at 17:34
add comment

I used the S3 REST/Curl API listed earlier in this thread and did this:

<?php
if (!class_exists('S3')) require_once 'S3.php';

// Instantiate the class
$s3 = new S3('accessKeyId', 'secretAccessKey');
S3::$useSSL = false;

// List your buckets:
echo "S3::listBuckets(): ";
echo '<pre>' . print_r($s3->listBuckets(), 1). '</pre>';

$totalSize = 0;
$objects = $s3->getBucket('name-of-your-bucket');
foreach ($objects as $name => $val) {
    // If you want to get the size of a particular directory, you can do
    // only that.
    // if (strpos($name, 'directory/sub-directory') !== false)
    $totalSize += $val['size'];
}

echo ($totalSize / 1024 / 1024 / 1024) . ' GB';
?>
share|improve this answer
add comment

... A bit late but, the best way I found is by using the reports in the AWS portal. I made a PHP class for downloading and parsing the reports. With it you can get total number of objects for each bucket, total size in GB or byte hrs and more.

Check it out and let me know if was helpful

AmazonTools

share|improve this answer
    
This is an interesting solution, although a little hackish. Worried about it breaking if/when Amazon changes their site, but I may have to try this out once I have enough objects that the other way becomes too slow. Another benefit of this approach is that you don't get charged for any API calls. –  Garret Heaton Dec 21 '09 at 16:16
    
. . . its an assumption but, if Amazon do change the look of their site, I doubt they would change the back end much, meaning the current GET and POST queries should work. I will maintain the class in the event it does break anyway as I use it often. –  Corey Sewell Dec 22 '09 at 0:26
add comment

If you download a usage report, you can graph the daily values for the TimedStorage-ByteHrs field.

If you want that number in GiB, just divide by 1024 * 1024 * 1024 * 24 (that's GiB-hours for a 24-hour cycle). If you want the number in bytes, just divide by 24 and graph away.

share|improve this answer
add comment

So trolling around through the API and playing some same queries, S3 will produce the entire contents of a bucket in one request and it doesn't need to descend into directories. The results then just requiring summing through the various XML elements, and not repeated calls. I don't have a sample bucket that has thousands of items so I don't know how well it will scale, but it seems reasonably simple.

share|improve this answer
    
This does seem to be the best option. Will update this post in the future if it scales poorly and I need to do something else. The library that ended up providing easy access to the raw API results was this PHP one: undesigned.org.za/2007/10/22/amazon-s3-php-class –  Garret Heaton Nov 16 '09 at 15:20
add comment

I wrote a Bash script, s3-du.sh that will list files in bucket with s3ls, and print count of files, and sizes like

s3-du.sh testbucket.jonzobrist.com
149 files in bucket testbucket.jonzobrist.com
11760850920 B
11485205 KB
11216 MB
10 GB

More info and full script here :

http://www.jonzobrist.com/2011/03/19/s3-du-sh-script-to-get-bucket-size-on-amazon-aws-s3/

It does do subdirectory size, as Amazon returns the directory name and the size of all of it's contents.

share|improve this answer
add comment

Also Hanzo S3 Tools does this. Once installed, you can do:

s3ls -s -H bucketname

But I believe this is also summed on the client side and not retrieved through the AWS API.

share|improve this answer
add comment

Hey there is a metdata search tool for AWS S3 at https://s3search.p3-labs.com/.This tool gives statstics about objects in a bucket with search on metadata.

share|improve this answer
add comment

By Cloudberry program is also possible to list the size of the bucket, amount of folders and total files, clicking "properties" right on top of the bucket

CloudBerry: http://www.cloudberrylab.com/

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.