I have a m1.small EC2 instance on AWS running some websites. I noticed my CPU usage have peaks at regular interval of times, exactly every 30 minutes (0:06, 0:36, 1:06, ...).

I've checked my crons (I have many), bot no one runs every 30 minutes. Looking at top I noticed that peaks are about about 1 minute long, and are almost entirely made of "stolen CPU" (%st). I've read that it's CPU time stolen by the Amazon VM hypervisor, but I can't understand why it happens (I'm not running CPU intensive stuffs when this occurs) and why it's exactly every 30 minutes.

Do you have any clue? Should I buy a bigger instance? I hope not, because the rest of the time CPU is very low and load average never goes over 0.5...

Cacti CPU graph

  • 1
    You realise it peaks at slightly less than 8% of total CPU, yes? Are you sure this is an actual problem? – MadHatter Mar 13 '14 at 16:55
  • @MadHatter On Cacti they are displayed at ~8% because of the 5 minute granularity. I intercepted those peaks with top and it's ~60% (almost all %st) for about 1 minute. – lorenzo-s Mar 13 '14 at 17:31
  • 1
    You're not even tracking steal time on this graph! – Michael Hampton Mar 13 '14 at 19:31
  • @MichaelHampton The Cacti graph is only to let you see the "regularity" of peaks. I used top to know that these peaks are in fact made of %st. – lorenzo-s Mar 14 '14 at 7:44
up vote 11 down vote accepted

Depending on the EC2 instance type and the underlying hardware, you may not be paying for access to all of the underlying CPU cycles. Amazon is not going to give you access to 100% of a modern, fast CPU if you have asked for an m1.small which is promised to be equivalent to an old, slow CPU.

On EC2, steal doesn't depend on the activity of other virtual machine neighbors. It is simply a matter of EC2 making sure you are not getting more CPU cycles than you are paying for.

If your m1.small gets 50% of the underlying faster CPU, then for every bit of CPU you are using, you will see another equal percentage flagged as steal.

It would be nice if EC2 let you think your true available CPU was "100%" instead of teasing you with the rest of the CPU that you don't have access to, and then telling you that you can't have it when you try to use the CPU, but that's the way it works given the current VM and host setup.

m1.small instances are likely to show a high percentage of steal given the limited CPU they have access to for the price compared to the CPU speeds on the underlying hardware.

If you are concerned that this particular instance might have something broken on EC2's side, you could stop/start it to move it to new hardware (my article on this) and see if that makes a difference. Of course, if the steal percentages drops, it might just indicate that you have moved to a slower hardware CPU.

As to the activity every 30 minutes, that is software on your server. It could be a system cron job or it could be triggered by a daemon (background process).

  • Eric, your answer is quite clear and informative, but I still don't understand why a lot of users (myself included) see and increase of stolen CPU % together with much slower response from the affected machine. If the VM always has access to the same "amount" of CPU and stolen CPU % "doesn't depend on the activity of other virtual machine neighbours", why do the VMs become slower as stollen CPU % goes up? – Miguel Ferreira Nov 8 '17 at 10:03

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.