28

Our infrastructure is on AWS. I want to get a daily report on how much spent on the previous day. What is the best way to do it?

2
  • Unless I'm mistaken, this question was asked 8 years ago and there still doesn't seem to be a way to simply set up a cloudwatch alarm if daily spend exceeds X amount. How the hell can this be?!!
    – rix
    Mar 9, 2020 at 13:23
  • @rix AWS Budgets does this since 2015. aws.amazon.com/blogs/aws/new-aws-budgets-and-forecasts
    – SeanFromIT
    Sep 3, 2020 at 23:04

12 Answers 12

21

Update

AWS has just announced the general availability of the functionality to Monitor Estimated Charges Using Billing Alerts via Amazon CloudWatch (it apparently has been available to AWS premium accounts already since end of 2011, see Daniel Lopez' answer to Is there a way to set Amazon AWS billing limit?):

We regularly estimate the total monthly charge for each AWS service that you use. When you enable monitoring for your account, we begin storing the estimates as CloudWatch metrics, where they'll remain available for the usual 14 day period. [...]

As outlined in the introductory blog post, You can start by using the billing alerts to let you know when your AWS bill will be higher than expected, see Monitor Your Estimated Charges Using Amazon CloudWatch for more details regarding this functionality.

This is already pretty useful for many basic needs, however, using the CloudWatch APIs to retrieve the stored metrics yourself (see GetMetricStatistics) actually allows you to drive arbitrary workflows and business logic based upon this data, and of course you could generate a daily report on how much spent on the previous day like so as well.

Regarding the latter, the scope of this offering is stressed as well though:

It is important to note that these are estimates, not predictions. The estimate approximates the cost of your AWS usage to date within the current billing cycle and will increase as you continue to consume resources. [...] It does not take trends or potential changes in your AWS usage pattern into account. [emphasis mine]

That is, the granularity of the reported metrics has yet to be analyzed (I see data points every 4 to 8 hours, but not necessarily updated values every time, as one would expect actually), so deriving a sufficiently precise daily report might require some statistical post processing.


Initial Answer

Unfortunately this is less straight forward than one would think, especially given that the desired data can be inspected manually via your account. There are two monitoring options one would expect:

  • notifications via email/RSS/etc.
  • API access to the data

Neither AWS nor any other IaaS/PaaS/SaaS vendor I'm aware of does offer API access to their accounting data currently (maybe due to the potential financial/legal implications), making any form of 3rd party integration (which would be easy to do nowadays) cumbersome at best, insofar you need to resort to web scraping to retrieve the data in the first place.

Fortunately a new offering from Cloudability [link removed after discontinuation of free tier] has entered the stage recently to do just this for you in a professional and vendor agnostic way, we are using it with great success already for AWS specifically - you'll currently receive a daily (or less frequent) report of your monthly spending only though, i.e. not broken down to your daily spending yet. Adding the daily increase would be trivial of course, so I'd hope and expect they'll make more information like this available over time.

Their approach to pricing [link removed after discontinuation of free tier] is refreshing as well (despite being obvious) and simply tied to your own cloud spending, thus should pay for itself as soon as you realize respective saving potential (they don't charge anything at all if you spend less than $2.5k/mo).
Update 20121016: Unfortunately Cloudability has changed their pricing model to a more common one, which still includes a free tier (and is reasonable priced in general), but removes access to advanced features therein, which I considered a refreshingly fair and smart approach for users with small budgets, who might still be multipliers elsewhere or upgrade once growing into it.

Update 20150115: Unfortunately Cloudability has chosen the path of many freemium SaaS vendors and finally discontinued the free tier entirely: From February 1, we will no longer offer the Cloudability Free edition that you are using today.

Update 20120427

The former caveat (kept for reference below) of requiring your main AWS credentials doesn't apply anymore - AWS recently introduced New IAM Features: Password Management and Access to Account Activity and Usage Reports Pages:

This new feature allows you to create separate and distinct IAM users for business and technical purposes. You can grant your business users access to the Account Activity and/or Usage Reports pages of the AWS website to allow them to access billing and usage data without giving them access to other AWS resources such as EC2 instances or files in S3

Cloudability has now integrated this as well, thus you don't need to hand them your main AWS credentials anymore or spent the extra effort to establish Consolidated Billing just to gain insight into your cloud spending, see How to Setup Amazon IAM (Identity Account Management) for details.

Former Caveat

There is one caveat one should be aware of upfront though:

In order to access your data you'll need to hand them your main AWS credentials, because otherwise they can't scrape your account, obviously. For AWS in particular you can still avoid this by facilitating Consolidated Billing, where you consolidate payment for multiple Amazon AWS accounts [...] by designating a single paying account, which in turn has no access to your computing resources and data.

2
  • I've just signed up in Cloudability but in their mail I could not find how much did we spent yesterday (or in last 24 hours). They are just giving monthly estimate and how much is spent till now.
    – Sabya
    Jan 18, 2012 at 14:45
  • 2
    @sabya - you are correct of course, I've only realized daily report and missed this additional requirement, sorry (I've updated the answer to reflect this)! Obviously it will be trivial for you to do the math yourself after the 2nd day, though having it available right in the mail would be an obvious improvement and much more convenient of course. Jan 18, 2012 at 15:57
15

Using awscli tools, you can get your month-to-date total:

$ aws --region us-east-1 cloudwatch get-metric-statistics \
    --namespace "AWS/Billing" \
    --metric-name "EstimatedCharges" \
    --dimension "Name=Currency,Value=USD" \
    --start-time $(date +"%Y-%m-%dT%H:%M:00" --date="-12 hours") \
    --end-time $(date +"%Y-%m-%dT%H:%M:00") \
    --statistic Maximum \
    --period 60 \
    --output text | sort -r -k 3 | head -n 1 | cut -f 2

2494.47

Totals from two different days can be subtracted to get the daily delta. Or, an estimate can be obtained in one go by increasing the time window (end-time - start-time) to 24h and subtracting the earliest data point from the latest.

Notes:

  • assumes you have enabled monitoring of your estimated charges
  • assumes you have GNU date (for other dates, e.g., on OSX, date -v-12H may work)
  • your Currency may vary
  • Amazon updates the CloudWatch metric every ~4h, so querying for a time window smaller than 4h may not return any data points. I've used a 12h window to be extra safe.
  • I wanted to include a link to the blog post on which my answer was based, but apparently I need 10 reputation points to post more than 2 links. :)
3
  • 4
    It's important to note that the only region that works for the 'AWS/Billing' namespace is us-east-1. I tore out much of my hair wondering why my account with access to billing was getting empty results via the CLI.
    – Corey Cole
    Jul 17, 2016 at 1:09
  • Just a note, that the awscli uses botocore and these attributes appear to be accessible via boto3.readthedocs.io/en/latest/reference/services/…
    – monkut
    Mar 22, 2017 at 6:36
  • So is there a way by now to make this work for other regions?
    – nh2
    Oct 5, 2017 at 15:12
4

If you really need a day-to-day cost report, you'll need to use the "Usage Report" tool in your AWS account. You can request a report for each service you use, on whatever time period you want, in granularity from by hour to by month. Then it downloads a CSV.

You'll need to do some post-processing on that CSV (since it's not in cost, but in usage etc), but it will provide you the data you need to have a day-to-day cost.

1
  • +1 Yes, I understand. I was just looking for an easier method.
    – Sabya
    Jan 19, 2012 at 4:57
3

Amazon provides your current month-to-date charges here:

http://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=activity-summary

Towards the top of the page it indicates how current the data is. I find it tends to lag by a few hours.

This is the most accurate and up-to-date record you can get from Amazon or anybody else at this time.

2

Maybe this Python module on Github can help you getting started: pyec2costs (for reserved or ondemand instances).

2

There is fairly new tool open-sourced by Netflix called Ice: https://github.com/Netflix/ice which allows to visualize the billing details as retrieved via the AWS reports generated into your S3 buckets.

2

Using CostExplorer AWS CLI command It is possible to retrive the cost and usage metrics for the Account

For Ex: If you need to retrive cost from starting of the month to present day you can use the below command.

aws ce get-cost-and-usage --time-period Start=$(date -u -d "$TODAY" '+%Y-%m-01'),End=$(date -u +"%Y-%m-%d" --date="+1 day") --granularity MONTHLY --metrics UnblendedCost --output text | sort -r -k 3 | head -n 1 | cut -f 2

Output: 615.6129260967

If you need to get the cost on DAILY basis use GRANULARITY as DAILY as shown below:

aws ce get-cost-and-usage --time-period Start=$(date -u -d "$TODAY" '+%Y-%m-01'),End=$(date -u +"%Y-%m-%d" --date="+1 day") --granularity DAILY --metrics UnblendedCost
1

Take a look at Xervmon. They provide day to day spend and usage in addition analytics on historical. They are an upcoming service provider with detailed integrations with Amazon AWS planned in next 3 months.

Some screenshots from my current account is as below. http://cloudmaya.net/~develop/demo/Slide1.jpg http://cloudmaya.net/~develop/demo/Slide2.jpg

Bunch of professionals have built and it is quite neat.

1

Here is a simple script that demonstrate how to parse and analyze your detailed AWS billing CSV file:

Should be easy enough so you can build your own analysis !

1

You can now do this with the Cost Explorer API. You can access very easily via awscli / boto3 for example but the point is that it's a lot easier than before.

0

I've seen companies build their own in-house tools for this - basically they scrap the AWS billing page and on their own dashboard, display the current cost, and in one example, they divide it by the days in the month that have passed, and multiple that to get the estimated total month cost.

AWS don't offer a billing API yet (I'm sure they will in future), but there are a couple of external services that can help. One is CloudVertical (disclosure: I work here), where you can get your daily, monthly, and hourly cost, broken down by service, and for multiple accounts.

The real holy grail for a service like AWS though is not just to track daily spending, but to show insights on efficiency (cost+usage = efficiency) and also highlight opportunities for savings (ie: times to use reserved or spot instances)

1
  • Ed Byme is correct. There are a number of providers. Cloudability is one as is CloudVertical. However they are all very heavy weight and expensive solutions with a lot of features you will probably never use. Instead, consider BillGist.com (disclosure: i work here) which is super cheap and emails you reports right in your inbox.
    – user256851
    Dec 3, 2014 at 7:04
0

There's is tool to view your AWS Billing Usage where you can slice and dice your usage across different AWS Resource to as low as 30 minutes. I've posted more thorough answer at https://stackoverflow.com/a/37585673/1526342

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .