11

Amazon Pricing on Spot Instance Inconsistencies

This is something which will be best explained through screenshots of a historical chart of instance pricings.

If you look at a lot of the instance prices for spot instances, you will notice regular patterns of spikes. See here:

enter image description here

As you can see, the price for this compute medium instance, regularly spikes above the on demand price.

A c1.medium instance (on demand), would only cost $0.186 per hour. But for a period of a few weeks, in zone B, the price would regularly spike to $1.20. This is some 6 times the actual on demand price.

It's also not isolated. If you look at zone-b again for small instances, there is a similar, spike frequently. Which goes 4x the on demand pricing.

enter image description here

Does anyone know why this happens?

Here are a few suggestions

  • Someone entered $1.2 instead of $0.12 (I would discount this since it happened 20 times over the space of 3 weeks).

  • Amazon regularly artifically inflate their prices by bidding on their own instances to get the most bang for their buck. (I would discount this since it would be ridiculous and bad business)

  • Some company launched 1000 servers at once, and wants to make sure that they all launch. (I would discount this since they would presumably launch them at a price which would be below the minimum on demand price. Why would you pay above on demand for a single server?).

  • It's a bug in their reporting?

2
  • There is a related question that may answer this question as well. Sep 10, 2012 at 2:20
  • I would not discount #2. "Bad business" would be if customers switched to the cheaper spot instances and didn't see the benefits of standard instances. Having instances be randomly terminated or paying for a large spike in cost is the price of using spot instances, whether or not this price is extracted by the hand of nature or by man. I don't think it would be below Amazon to step in if the market wasn't working exactly as they wanted (whether to set the price directly or to create shortages). Jul 30, 2014 at 15:37

4 Answers 4

6

There is a paper by folks from Israel Institute of Technology which analyzes the spot market. Their conclusion is that spot prices are not bid driven but rather generated with AR(1) function. So in such case it wouldn't be due to other's bids, but due to algorithm used.

http://www.cs.technion.ac.il/~ladypine/spotprice-ieee.pdf

Though, I guess it could have changed if spot market really reached the level where they could get nice prices from demand/supply. As Matt suggest it become a common practice to drive costs down with spot instances and many overbid on-demand prices to avoid down time. In longer run it was still cost effective ... but is it still? The real problem is that apart of Amazon nobody really knows how the prices are created.

1
  • 8
    THIS IS NOT THE CORRECT ANSWER! 1. The OP is asking about the three months prior to Sept 2012; the research data only goes up to 2011. 2. The authors have said that as of Nov 2011 they no longer observe the AR(1) behaviour, and in fact on of the original authors of the paper has said so in an answer to this very question! 3. An AR(1) function is not spiky. 4. MOST IMPORTANTLY: The analysis was done for an interval of time specifically chosen to not have spikes! Apr 20, 2013 at 5:55
9

This is not a bug, this is an priority / availability issue because on-demand instances have priority over spot-instances regardless of spot-bids. This stems from Amazon EC2 statement that you "bid on spare Amazon EC2 instances".

So when the demand for on-demand instances increases there are less spares available, and the availability of spot-instances decreases driving spot pricing higher according to competing spot-bids.

Many spot-bidders have failed to understand the higher priority of on-demand instances and have bid above on-demand pricing in an attempt to gain priority over on-demand instance requests, but this fails because on-demand instances have higher priority regardless of spot biding prices.

In turn these above-the-on-demand pricing bids have been competing against each other driving the highest prices multiple times higher than on-demand pricing when spot-instances availability is low.

Although the following was not asked, my recommendations for lowest-cost spot instances management of long-term workloads is:

  1. Design your application for failures (continuous backup / replication), and rapid recovery
  2. Set spot-instance bid price to that of on-demand pricing
  3. When a spot instance is terminated, restart it as an on-demand instance or in another region/zone where spot pricing is lower.
  4. Monitor spot-pricing until it goes lower than on-demand pricing
  5. Stop on-demand instance and goto 2. above
8

EC2 no longer use the artificial AR(1) process to control the minimal price. Even when they did, they only changed the minimal price - the AR(1) process was anchored, and never caused prices to jump above the on-demand price. Such jumps can come either from users specifying high prices (possibly more than they really intended to pay), or from Amazon wishing to shut down a portion of the spot instances, like it seems they did for m1.small in the summer of 2010.

You can read more in the journal version of our paper, in the last section which is more updated (the postmortem section):

4

It's not a bug in the reporting. Sometimes, the spot prices do spike. In the past, they've spiked to enormous amounts. Since then, Amazon has improved the algorithm in which spot instances are bid on and allocated, but even still, spot prices do spike from time to time.

Back in September 2011, spot prices for m2.2xlarge in one us-east-1 AZ spiked to $999/hour.

The reason is simple: people are specifying maximum spot prices that are higher than the on-demand price, and demand is pushing the prices that high.

Many people use spot instances as a way to minimize the cost of the running EC2 instances. Quite often, they will specify high maximum spot prices in order to reduce the chances that their spot instance will be out-bid and terminated. It's a gamble that may pay off in the long-run, but runs the risk of getting very expensive very fast if they're not careful.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .