3

I have an Amazon EC2 instance (c1.medium, ami-ed46a784) that experiences short spikes in load average every 32 hours or so. The average load is normally around 0.15 but rises to 3+ for about 15 minutes during these spikes. During this time there is no increase in CPU usage, disk traffic, swap usage, IRQ interrupts, apache traffic, or any other metric that my collectd installation is reporting. Disabling all cron jobs I'd added to the base install had no effect. This instance runs chef, apache, mysql, couchdb, memcached, and a twisted python service - all of which currently receive very little traffic.

Below is some data collected during one of these spikes:

/usr/bin/top output: 
top - 14:31:00 up 65 days, 20:48, 1 user, load average: 3.00, 2.13, 1.01 
Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie 
Cpu(s): 0.8%us, 0.5%sy, 0.0%ni, 98.0%id, 0.2%wa, 0.0%hi, 0.1%si, 0.4%st 
Mem: 1788724k total, 1723448k used, 65276k free, 179284k buffers 
Swap: 917496k total, 124k used, 917372k free, 680404k cached 
...process with the most CPU usage has just 4%... 
/usr/bin/iostat output: 
Linux 2.6.21.7-2.fc8xen (foo.example.com) 11/08/09 _i686_ (2 CPU) 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          0.78    0.00    0.80    0.19    0.42   97.95

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda2              0.00         0.00         0.01       8762      74472
sda3              0.00         0.00         0.00        944        288
sda1             19.29         0.24       242.63    1354218 1380794096
sdb               0.34         0.07         6.50     393928   36997032
/usr/bin/mpstat output: 
Linux 2.6.21.7-2.fc8xen (foo.example.com) 11/08/09 _i686_ (2 CPU) 
14:31:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 
14:31:00 all 0.78 0.00  0.51 0.19    0.00 0.14  0.42   0.00   97.95 
/usr/bin/free -m output: 
     total used free shared buffers cached 
Mem: 1746  1683 63   0      175     664 

collectd load graph

Is it possible that someone else's instance on the same physical host could cause these spikes? Is there any other data I should try collecting during a spike that would help diagnose the issue? What else can contribute to load average?

This issue is also posted on the AWS forum.

Update #1

I also gathered data using dstat and published it here. It doesn't show anything causing load as far as I can tell.

1 Answer 1

1

I would check the cronjobs running on your host. In fact if you have a virt and on the same physical host there is another virt using for example the sata drives bandwidth and you want to write in the same time to the disk, it can cause bigger load then usual it would be on a single non-virtualized, non-shared environment. It is true for any IO operation. Btw. would you mind to press 1 when you are in top just to see all the cores. It seems there is no real load on your host at least there is no IOwait or any kind of reason why we can see thos 3.00 load. I am curious what can you see on both cores you have. Also if you can install dstat and run it to check what is going on.

2
  • Thanks for the suggestions. The next time it happens I'll capture the individual cpu states in top as well as have full dstat data. dstat looks like a great tool!
    – powdahound
    Nov 10, 2009 at 16:10
  • I added a link to the dstat data up above. Doesn't seem to show anything causing the load.
    – powdahound
    Nov 11, 2009 at 15:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .