4

I have a m1.small EC2 instance on AWS running some websites. I noticed my CPU usage have peaks at regular interval of times, exactly every 30 minutes (0:06, 0:36, 1:06, ...).

I've checked my crons (I have many), bot no one runs every 30 minutes. Looking at top I noticed that peaks are about about 1 minute long, and are almost entirely made of "stolen CPU" (%st). I've read that it's CPU time stolen by the Amazon VM hypervisor, but I can't understand why it happens (I'm not running CPU intensive stuffs when this occurs) and why it's exactly every 30 minutes.

Do you have any clue? Should I buy a bigger instance? I hope not, because the rest of the time CPU is very low and load average never goes over 0.5...

Cacti CPU graph

Improve this question
4
  • 1
    You realise it peaks at slightly less than 8% of total CPU, yes? Are you sure this is an actual problem?
    – MadHatter
    Mar 13, 2014 at 16:55
  • @MadHatter On Cacti they are displayed at ~8% because of the 5 minute granularity. I intercepted those peaks with top and it's ~60% (almost all %st) for about 1 minute.
    – lorenzo-s
    Mar 13, 2014 at 17:31
  • 1
    You're not even tracking steal time on this graph!
    – Michael Hampton
    Mar 13, 2014 at 19:31
  • @MichaelHampton The Cacti graph is only to let you see the "regularity" of peaks. I used top to know that these peaks are in fact made of %st.
    – lorenzo-s
    Mar 14, 2014 at 7:44

1 Answer 1

Reset to default
12

Depending on the EC2 instance type and the underlying hardware, you may not be paying for access to all of the underlying CPU cycles. Amazon is not going to give you access to 100% of a modern, fast CPU if you have asked for an m1.small which is promised to be equivalent to an old, slow CPU.

On EC2, steal doesn't depend on the activity of other virtual machine neighbors. It is simply a matter of EC2 making sure you are not getting more CPU cycles than you are paying for.

If your m1.small gets 50% of the underlying faster CPU, then for every bit of CPU you are using, you will see another equal percentage flagged as steal.

It would be nice if EC2 let you think your true available CPU was "100%" instead of teasing you with the rest of the CPU that you don't have access to, and then telling you that you can't have it when you try to use the CPU, but that's the way it works given the current VM and host setup.

m1.small instances are likely to show a high percentage of steal given the limited CPU they have access to for the price compared to the CPU speeds on the underlying hardware.

If you are concerned that this particular instance might have something broken on EC2's side, you could stop/start it to move it to new hardware (my article on this) and see if that makes a difference. Of course, if the steal percentages drops, it might just indicate that you have moved to a slower hardware CPU.

As to the activity every 30 minutes, that is software on your server. It could be a system cron job or it could be triggered by a daemon (background process).

Improve this answer
1
  • 2
    Eric, your answer is quite clear and informative, but I still don't understand why a lot of users (myself included) see and increase of stolen CPU % together with much slower response from the affected machine. If the VM always has access to the same "amount" of CPU and stolen CPU % "doesn't depend on the activity of other virtual machine neighbours", why do the VMs become slower as stollen CPU % goes up?
    – Miguel Ferreira
    Nov 8, 2017 at 10:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .