AWS S3 charging for 4 TB of storage when only using less than 1 GB

Question

I'm having trouble understanding my large S3 bill, and figured I'd ask here before dropping $30 on AWS monthly support.

Basically, I have an Amazon EC2 instance that makes an API to different cryptocurrency exchanges and saves the responses to the instance HD. Calls are made about every 5 minutes, response objects are about 100 kb, is read by an R script, and added to a CSV file every ~8 minutes. That CSV file is synchronised to an Amazon S3 bucket about every 15 minutes.

The CSV files are usually 10 MB or so, for about 15 cryptocurrencies, every 15 minutes. So looking in the Amazon S3 bucket, there might be 0.5 GB of space used at the most.

However, the 'TimedStorage-ByteHours' reads at about 4 TB!

Amazon Simple Storage Service TimedStorage-ByteHrs $89.55

$0.000 per GB - storage under the monthly global free tier5 GB - Mo $0.00

$0.023 per GB - first 50 TB / month of storage used 3,893.399 GB - Mo $89.55

Any ideas?

Answer 1

Most likely you've got S3 Versioning enabled - that means old objects when overwritten with a newer version don't get deleted but are instead hidden in a history. To verify go to the Bucket -> Properties -> Versioning.

You can also view the old versions in the browser, like on this screenshot I've got several versions of the 108c05...json file:

If you've got versioning enabled but don't want to you can Suspend versioning but be aware that it won't delete the old versions, you'll have to either:

• use AWS-CLI and some scripting (start with aws s3api list-object-versions)
• configure Bucket Lifecycle Policy to expire the old versions. That's done through S3 -> bucket -> Management -> Lifecycle -> Add lifecycle rule and then on the Expiration screen fill these details:

Hope that helps :)

Answer 2

Infrequent-Access Storage has 30 Days minimum charge per Item

1. Go to https://s3.console.aws.amazon.com and get the file listing for one of your buckets.
2. Note that each file in the bucket has a storage class. The choice of storage class is under your control and affects costs.

Standard is the most expensive, so some people use infrequent access which has a lower monthly fee (~50% discount) per GB stored according to the table of prices. BUT, Infrequent also has a $0.01/GB retrieval charge, so only saves on total costs for files stored at least one month and retrieved less than once per month. Also, there can be a lifecycle policy on your bucket that changes the storage class of files over time, which can affect when the 30-day minimum storage period kicks in.

In the fine print, infrequent access has a minimum charge of 30 days for each item stored.

Source: https://aws.amazon.com/s3/pricing/

First footnote, marked †

† S3 Standard-Infrequent Access and S3 One Zone-Infrequent Access storage have a minimum billable object size of 128KB. Smaller objects may be stored but will be charged for 128KB of storage. S3 Standard-Infrequent Access and S3 One Zone-Infrequent Access Storage are charged for a minimum storage duration of 30 days. Objects that are deleted, overwritten, or transitioned to a different storage class before 30 days will incur the normal usage charge plus a pro-rated request charge for the remainder of the 30 day minimum. Objects stored 30 days or longer will not incur a 30-day minimum request charge

What's expensive about Amazon S3, and Google Cloud Storage as well, are the network charges if data is sent out onto the public internet instead of kept in that company's cloud. The network charge for retrieval one time can easily exceed the storage charge for 3-4 months. $0.09/GB (Oct. 2018) for AWS S3, $0.12/GB for Google Cloud Storage, lowest usage tier.

Digital Ocean has an S3 compatible storage layer called "Spaces" that might be worth checking out. Some bugs have been reported, and it might be throttled in some ways, but the overall pricing looks interesting and it might be "good enough" for some uses.