4

I save user profile pictures on Amazon S3 with some random path/filenames for each user. The database knows the path for each user image. My question is that when user navigates within the app, a get request is issued to the profile picture several times. After the first time, S3 returns http 304 Not Modified status. Does every such get request count against the GET request quota for billing?

7
  • I'd contact AWS Support on this one.
    – ceejayoz
    Apr 21, 2018 at 14:08
  • "a get request is issued" - Why should a GET request with the HTTP response status code "302 Found" not count as a GET request for "Request Pricing"?
    – Fabian
    Apr 21, 2018 at 15:51
  • 1
    calculator.s3.amazonaws.com/index.html (there is no pricing model based on response status code)
    – Fabian
    Apr 21, 2018 at 16:33
  • 1
    There is a problem here with the premise of the question. "After the first time, S3 returns http 302 Found status". No, it doesn't... or, it shouldn't. 302 Found is a temporary redirect. I believe you have made an incorrect observation or perhaps drawn an incorrect conclusion from observing the behavior of a very new bucket (created in the last few minutes or hours) which might return some 302's early in its life, depending on how you access it, but these would be the first request, followed by the browser making a correct request to the regional endpoint and getting a 200 or 304. Apr 21, 2018 at 20:23
  • 1
    @Michael-sqlbot I think I made a terrible mistake by mentioning 302 Found. In fact, I intended to say 304 Not Modified. I am editing the question to fix this. However, based on the comments, I think it will be chargeable no matter if server returned a 200 or 304. Thanks for your inputs. Apr 22, 2018 at 16:26

2 Answers 2

7
+100

I understand that you're looking for firm proof as to whether GET requests that return a 304 are counted as a GET request for billing purposes, in the same way that a GET request returning a 200 is.

The documentation does not call out any billing difference based on the returned HTTP Status Code, but this does not offer any proof one way or another. To provide you with a firm answer, I setup a Cost and Usage Report for my account with the aggregation frequency set to hourly and resource ids included, and then tested this and looked at what appeared in the report.

TLDR; The billing report confirms what everyone expected, a GET is billed as a GET whether it returns a status code of 200 or 304.

Test Setup

I selected an unused publically readable S3 bucket of mine and uploaded an image file and set an HTTP Header of Cache-Control: public, max-age=10 on the file. This means if the page is reloaded after 10 seconds then the browser must check with the server to see if the file has changed before using its locally cached copy.

To ensure the test would be repeatable and not have any unexpected results (e.g. from a typo etc), I created a local html file that would display the image.

<html>
<body>
<img src="http://mybucket.s3.amazonaws.com/5DS05736-2.jpg" width="868" height="488" />
</body>
</html>

The reason for keeping this on my local machine was to avoid any requests to S3 for things like favicon (which would potentially pollute the results).

Confirming the test would work

I performed an initial test in chrome by opening developer tools, looking in the network tab and loading my local html file. On the first request the image was loaded with an HTTP Status Code of 200. Waiting and then refreshing the page showed the image being loaded a second time with an HTTP Status Code of 304.

I cleared the cache of entries from the last hour.

Actual Test

The next morning I repeated the experiment. To make sure I could distinguish between tests from the previous evening and the real test, I noted the time and the status codes I got. I did one load without the image in the cache resulting in a 200, and then two refreshes ~20 seconds apart resulting in 304s.

This would give the following possibilities in the billing report:

  • 1 GET Request = Only GET requests returning 200 are counted
  • 2 GET Requests = Only GET requests returning 304 are counted (unlikely)
  • 3 GET Requests = All GET requests are counted

The Billing Report

I then waited until the billing report was available and loaded it into Athena. If you're repeating this, the Athena CREATE TABLE command is:

CREATE EXTERNAL TABLE `cost_and_usage`(
  `identity_lineitemid` string, 
  `identity_timeinterval` string, 
  `bill_invoiceid` string, 
  `bill_billingentity` string, 
  `bill_billtype` string, 
  `bill_payeraccountid` string, 
  `bill_billingperiodstartdate` string, 
  `bill_billingperiodenddate` string, 
  `lineitem_usageaccountid` string, 
  `lineitem_lineitemtype` string, 
  `lineitem_usagestartdate` string, 
  `lineitem_usageenddate` string, 
  `lineitem_productcode` string, 
  `lineitem_usagetype` string, 
  `lineitem_operation` string, 
  `lineitem_availabilityzone` string, 
  `lineitem_resourceid` string, 
  `lineitem_usageamount` string, 
  `lineitem_normalizationfactor` string, 
  `lineitem_normalizedusageamount` string, 
  `lineitem_currencycode` string, 
  `lineitem_unblendedrate` string, 
  `lineitem_unblendedcost` string, 
  `lineitem_blendedrate` string, 
  `lineitem_blendedcost` string, 
  `lineitem_lineitemdescription` string, 
  `lineitem_taxtype` string, 
  `product_productname` string, 
  `product_accountassistance` string, 
  `product_architecturalreview` string, 
  `product_architecturesupport` string, 
  `product_availability` string, 
  `product_bestpractices` string, 
  `product_cacheengine` string, 
  `product_caseseverityresponsetimes` string, 
  `product_clockspeed` string, 
  `product_currentgeneration` string, 
  `product_customerserviceandcommunities` string, 
  `product_databaseedition` string, 
  `product_databaseengine` string, 
  `product_dedicatedebsthroughput` string, 
  `product_deploymentoption` string, 
  `product_description` string, 
  `product_durability` string, 
  `product_ebsoptimized` string, 
  `product_ecu` string, 
  `product_endpointtype` string, 
  `product_enginecode` string, 
  `product_enhancednetworkingsupported` string, 
  `product_executionfrequency` string, 
  `product_executionlocation` string, 
  `product_feecode` string, 
  `product_feedescription` string, 
  `product_freequerytypes` string, 
  `product_freetrial` string, 
  `product_frequencymode` string, 
  `product_fromlocation` string, 
  `product_fromlocationtype` string, 
  `product_group` string, 
  `product_groupdescription` string, 
  `product_includedservices` string, 
  `product_instancefamily` string, 
  `product_instancetype` string, 
  `product_io` string, 
  `product_launchsupport` string, 
  `product_licensemodel` string, 
  `product_location` string, 
  `product_locationtype` string, 
  `product_maxiopsburstperformance` string, 
  `product_maxiopsvolume` string, 
  `product_maxthroughputvolume` string, 
  `product_maxvolumesize` string, 
  `product_maximumstoragevolume` string, 
  `product_memory` string, 
  `product_messagedeliveryfrequency` string, 
  `product_messagedeliveryorder` string, 
  `product_minvolumesize` string, 
  `product_minimumstoragevolume` string, 
  `product_networkperformance` string, 
  `product_operatingsystem` string, 
  `product_operation` string, 
  `product_operationssupport` string, 
  `product_physicalprocessor` string, 
  `product_preinstalledsw` string, 
  `product_proactiveguidance` string, 
  `product_processorarchitecture` string, 
  `product_processorfeatures` string, 
  `product_productfamily` string, 
  `product_programmaticcasemanagement` string, 
  `product_provisioned` string, 
  `product_queuetype` string, 
  `product_requestdescription` string, 
  `product_requesttype` string, 
  `product_routingtarget` string, 
  `product_routingtype` string, 
  `product_servicecode` string, 
  `product_sku` string, 
  `product_softwaretype` string, 
  `product_storage` string, 
  `product_storageclass` string, 
  `product_storagemedia` string, 
  `product_technicalsupport` string, 
  `product_tenancy` string, 
  `product_thirdpartysoftwaresupport` string, 
  `product_tolocation` string, 
  `product_tolocationtype` string, 
  `product_training` string, 
  `product_transfertype` string, 
  `product_usagefamily` string, 
  `product_usagetype` string, 
  `product_vcpu` string, 
  `product_version` string, 
  `product_volumetype` string, 
  `product_whocanopencases` string, 
  `pricing_leasecontractlength` string, 
  `pricing_offeringclass` string, 
  `pricing_purchaseoption` string, 
  `pricing_publicondemandcost` string, 
  `pricing_publicondemandrate` string, 
  `pricing_term` string, 
  `pricing_unit` string, 
  `reservation_availabilityzone` string, 
  `reservation_normalizedunitsperreservation` string, 
  `reservation_numberofreservations` string, 
  `reservation_reservationarn` string, 
  `reservation_totalreservednormalizedunits` string, 
  `reservation_totalreservedunits` string, 
  `reservation_unitsperreservation` string, 
  `resourcetags_username` string, 
  `resourcetags_usercostcategory` string)
ROW FORMAT DELIMITED 
  FIELDS TERMINATED BY ',' 
  LINES TERMINATED BY '\n' 
WITH SERDEPROPERTIES ( 
  'escape.delim'='\\') 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  's3://my-billing-bucket/reports/hourly/20180501-20180601/dcd20d15-fffd-4a40-bc5d-555f32fc64b2'
TBLPROPERTIES (
  'skip.header.line.count'='1')

And the query to narrow it down to the usage you're looking for is:

select * from cost_and_usage where lineitem_resourceid = 'my-bucket' and lineitem_operation = 'GetObject' and lineitem_usagetype = 'USW2-Requests-Tier2' order by lineitem_usagestartdate desc

You will need to change my-bucket and USW2-Requests-Tier2 to the appropriate values for your bucket and bucket location etc.

For the hour in which I performed my test, the lineitem_usageamount column lists 3 requests. This means that all of my requests were billed.

If you're wondering how I can be sure no other use of the bucket happened, I have CloudTrail with S3 logging enabled and pointing at CloudWatch Logs. Using this I was able to verify that no other requests were made to the bucket at that time. :)

1
  • 1
    It's amazing to see the time and effort you put in for performing the tests and coming up with a solid proof. You're the man! May 6, 2018 at 9:19
1

Does every such get request count against the GET request quota for billing?

Yes. "GET, SELECT and all other Requests" are in the exact same price category (source: Amazon S3 Pricing). The 302 requests are part of GET category (talking about "Request Pricing", your question is not about "Data Transfer Pricing").

4
  • I'm not sure this answers the question. It's not clear from that documentation that one of AWS's built-in 302s counts.
    – ceejayoz
    Apr 21, 2018 at 14:08
  • Maybe. There are 302s the user sets up, but there are 302s issued automatically by Amazon (like when you hit a non-us-east-1 bucket on the s3.amazonaws.com URL) that may not count against the bucket's requests count. I'd want a clear answer from AWS if OP's going to be making billions of such requests.
    – ceejayoz
    Apr 21, 2018 at 14:16
  • Maybe not billions, but as user base grows and they navigate across various pages in application, the number can become big and will eventually affect pricing. So just wanted to know how it works. Apr 21, 2018 at 15:19
  • 1
    This is the correct answer. Amazon does not differentiate based on request return code. "304 Not Modified" is a return code for GET requests. Amazon spent the computing resources to get your request, check it against metadata and determine your file has not changed so it'll be charged normally. May 3, 2018 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.