4

It seems like I can get total cost of S3 using the service level in the filter.

However I want to get cost per bucket level.

Can it be done in Cost Explorer?

If not, can I get the breakdown with aws cli?

enter image description here

3 Answers 3

6

I'd look into using cost allocation tags:

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html https://docs.aws.amazon.com/AmazonS3/latest/dev/CostAllocTagging.html

You'll need to Activate cost allocation tags under your billing dashboard, then add the tags you wish to track (ie billinggroup).

Then Under each bucket -> properties -> Advanced Setting -> Tags set a tag like billinggroup: bucketname

You could also do something like this with aws cli, but I don't think it will be as accurate, since it wont calculate the size of all object versions and replicas, but it might work for a ball park figure:

bucketlist="mybucket1 mybucket2"
echo -e "Bucket\tObjectCount\tTotalSize\n"
for x in ${bucketlist}; do 
    echo -en "$x\t"; aws s3 ls s3://$x --recursive | grep -v -E "(Bucket: |Prefix: |LastWriteTime|^$|--)" | awk 'BEGIN {count=0; total=0}{count++; total+=$3}END{print count"\t"total" ("total/2^30"GB)\t$"(total/2^30)*0.023}'
done
1
  • 4
    I've always wanted to name a variable bucketlist.
    – Underyx
    Jun 21, 2022 at 19:48
4

AWS cover this in their knowledge base: https://aws.amazon.com/premiumsupport/knowledge-center/s3-find-bucket-cost/

Basically, you need to add a tag to every bucket and set the value to be the bucket name. Then you can use the cost explorer to find that total cost per bucket:

screenshot of aws cost report

If you have lots of buckets, this is very tedious and error-prone so I wrote a Python script to automate the task:

import boto3
from botocore.exceptions import ClientError

s3_client = boto3.client('s3')
s3_resource = boto3.resource('s3')

def add_bucket_name_tag_to_all_buckets():
    TAG_NAME = 's3-bucket-name'

    for s3_bucket in s3_resource.buckets.all():
        s3_bucket_name = s3_bucket.name
        print(f'Setting tag "{TAG_NAME}" in "{s3_bucket_name}" to "{s3_bucket_name}"...')

        # Create tag iff there are no tags at all
        bucket_tagging = s3_resource.BucketTagging(s3_bucket_name)
        try:
            tags = bucket_tagging.tag_set # This throws if there are no tags
        except ClientError:
            tags = [{'Key':TAG_NAME, 'Value': s3_bucket_name}]
            bucket_tagging.put(Tagging={'TagSet':tags}) # Use carefully, this overwrites all tags

        # Now append tag if not present
        tags = bucket_tagging.tag_set
        if len([x for x in tags if x['Key'] == TAG_NAME]) == 0: # If tag not found, append it
            tags.append({'Key':TAG_NAME, 'Value': s3_bucket_name})
            bucket_tagging.put(Tagging={'TagSet':tags}) # Use carefully, this overwrites all tags


if __name__ == '__main__':
  add_bucket_name_tag_to_all_buckets()

This will recurse through your buckets and append a new tag called s3-bucket-name and set the value to be that name of that bucket.


If you don't understand Python, or trust running scripts from the internet(!), you can just do it manually as per the Amazon instructions, pasting contents below:


To check which Amazon S3 bucket is increasing your storage cost, perform the following steps:

  1. Add a common tag to each bucket.

  2. Activate the tag as a cost allocation tag.

Important: All tags can take up to 24 hours to appear in the Billing and Cost Management console.

  1. Use the AWS Cost Explorer to create an AWS Cost and Usage Report for your tag.

Note: Cost allocation tags don't show you costs that you incurred before you set up the tags.

Resolution Before you begin, your AWS Identity and Access Management (IAM) policy must have permissions to do the following:

Access the Billing and Cost Management console Perform the actions s3:GetBucketTagging and s3:PutBucketTagging Tip: Avoid using your AWS account root user for this solution. Instead, use an IAM user or role with the permissions that you need.

Adding a common tag to each bucket

  1. Open the Amazon S3 console.

  2. From the list of buckets, choose the bucket that you want to track costs for.

  3. Choose the Properties view.

  4. Scroll down and choose Tags.

  5. Choose Edit.

  6. Choose Add Tag.

  7. For Key, enter a name for the tag that you'll add to all the buckets that you want to track costs for. For example, enter "S3-Bucket-Name".

  8. For Value, enter the name of the bucket.

  9. Repeat Steps 1 through 7 for all the buckets that you want to track costs.

Activating the tag as a cost allocation tag

  1. Open the Billing and Cost Management console.

  2. From the navigation pane, choose Cost allocation tags.

  3. In the search bar, enter the name of the tag that you created for your buckets. For example, type "S3-Bucket-Name".

  4. Select the tag.

  5. Choose Activate.

Use the AWS Cost Explorer to create a cost report for the tag

  1. Open the Billing and Cost Management console.

  2. From the navigation pane, choose Cost Explorer.

  3. Choose Launch Cost Explorer.

  4. From the navigation pane, choose Reports.

  5. Choose New report.

  6. For Report Templates, select Cost & Usage report, and then choose Create Report.

  7. Under Filters, for Service, select S3 (Simple Storage Service). Then, choose Apply filters.

  8. For Tag, select the tag that you created. For example, select S3-Bucket-Name. Then, check each bucket that you want to track costs on, and choose Apply filters.

Note: If you don't see your tag in the filter list, it's likely that the tag was recently created and applied to a bucket. Wait 24 hours, and then try creating the report again.

  1. Under Advanced options, confirm that Show only untagged resources is unchecked.

  2. From the top of the graph, choose Group by, and then select the tag that you created.

  3. Choose Save as.

  4. Enter a title for your cost report.

  5. Choose Save Report.

After you create the cost report, use the report to review the cost of each bucket marked with the cost allocation tag that you created.

Note: You can set up a daily or hourly AWS Cost and Usage Report report to get more Amazon S3 billing details. However, these reports don't show you who made requests to your buckets. To see where requests to your bucket are coming from, enable either object-level logging or server access logging. To get more information on certain Amazon S3 billing items, you must enable logging ahead of time. Then, you'll have logs that contain Amazon S3 request details.

1
  • This doesn't handle S3 buckets in different locations. Feb 7 at 4:50
0

Python script I used to loop through each of the buckets with the tag name s3-bucket-name and then save the results to a csv file.

import boto3
import csv
from datetime import datetime, timedelta

# Set up AWS credentials and region
session = boto3.Session(
    aws_access_key_id='YOUR_ACCESS_KEY_ID',
    aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',
    region_name='YOUR_REGION'
)

# Set up S3 client and CloudWatch client
s3 = session.client('s3')
cloudwatch = session.client('cloudwatch')

# Set the date range for the cost report
start_time = datetime.utcnow() - timedelta(days=30)
end_time = datetime.utcnow()

# Set the tag key to filter by
tag_key = 's3_bucket_name'

# Get the S3 bucket names that have the specified tag
bucket_names = []
response = s3.list_buckets()
for bucket in response['Buckets']:
    response = s3.get_bucket_tagging(Bucket=bucket['Name'])
    for tag in response['TagSet']:
        if tag['Key'] == tag_key:
            bucket_names.append(bucket['Name'])

# Get the S3 bucket usage metrics for the specified date range and tag
costs_and_sizes_per_bucket = {}
for bucket_name in bucket_names:
    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/S3',
        MetricName='BucketSizeBytes',
        Dimensions=[{'Name': 'BucketName', 'Value': bucket_name}],
        StartTime=start_time,
        EndTime=end_time,
        Period=86400,
        Statistics=['Average']
    )
    if 'Datapoints' in response:
        for datapoint in response['Datapoints']:
            bucket_size = datapoint['Average'] / 1024 / 1024 / 1024
            cost = datapoint['Average'] * 0.000000001 * 0.023
            costs_and_sizes_per_bucket[bucket_name] = {
                'size': bucket_size,
                'cost': costs_and_sizes_per_bucket.get(bucket_name, {}).get('cost', 0) + cost
            }

# Save the bucket name, size, and cost in a CSV file
with open('s3_costs.csv', mode='w', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(['Bucket Name', 'Size (GB)', 'Cost'])
    for bucket_name, data in costs_and_sizes_per_bucket.items():
        size = data['size']
        cost = data['cost']
        writer.writerow([bucket_name, size, cost])

print('CSV file saved successfully!')
1
  • For Daniel's answer above, I had to add { 'Name': 'StorageType', 'Value': 'StandardStorage' } as a dimension for this to pull data back from cloudwatch.
    – joe
    Aug 11, 2023 at 16:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .