up vote 2 down vote favorite
1
share [g+] share [fb]

For the last two weeks we are having intermittent severe spikes in system cpu usage (shown as %sys), which last for maybe half a minute, locking most processes, including ssh.

I've been trying to figure this out, but atop doesn't show anything relevant (system usage for processes it shows is insignificant), spikes are intermittent and I could not reproduce the spike using any workload for the web application this webserver hosts.

If you have any ideas on how to debug high %sys and (sometimes) %si CPU usage, please share them.

System specs (don't know if any of this is relevant): Dedicated server, CentOS 6, core i7 950, consistent 4 to 8 GB RAM free at any time, hard drives are in RAID-1.

Additional info:

  • dmesg output doesn't change between spikes
  • /var/log/messages doesn't change between spikes
  • Here is cat /proc/vmstat
  • Here is output of mpstat 1 during a typical spike

Add 07.11.11: looks like simple reboot restored system state, and we might never know what caused the disturbance in first place.

link|improve this question
You could put some files onto a webpage from the time where you see that high load or locks: screenshot of top, dmesg and/or /var/log/syslog, /proc/vmstat. You could remove sensitive data before if needed. – ott-- Nov 3 '11 at 17:45
@ott-- added more info to the first post. – Mark Nov 3 '11 at 19:20
Apparently, I can't add any more links to the post without being considered a spammer, here is output of iostat -x 5 during a typical spike. – Mark Nov 3 '11 at 19:31
Do you run a BTRFS filesystem on Linux 3.0? – mailq Nov 3 '11 at 22:01
@mailq: No, Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux, file systems are all ext3. – Mark Nov 4 '11 at 5:18
feedback

2 Answers

up vote 0 down vote

High %si would suggest a high interrupt rate (si is the time spent in softirq handlers, AFAIK). Therefore my first guess would be that the server network interface is being hammered.

link|improve this answer
How can I prove or disprove this hypothesis? It does not seem like so, but it might be. – Mark Nov 3 '11 at 19:25
That %si corresponds to time spent in softirq handlers? Well, if you don't believe me or some other documentation you might find, you can read the kernel source code. – janneb Nov 4 '11 at 7:04
Err, no, hypothesis about network interface being hammered. – Mark Nov 4 '11 at 7:09
Ah, you can check e.g. /proc/softirq; unfortunately I don't know of any tool that displays individual softirqs over time. Alternatively, run something like "dstat -ar --socket --tcp" when you get a spike and post the results. – janneb Nov 4 '11 at 8:33
thanks a lot for dstat tool, it's kinda cool. Unfortunately, it's still not obvious, what causes spikes, see it for yourself: pastie.org/pastes/2821888/text?key=yqtv1iulh9nyhgahod1eq – Mark Nov 6 '11 at 19:58
feedback
up vote 0 down vote accepted

It sounds stupid, but reboot helped and we might never know, what caused the spikes in first place.

Thank you for responses, though.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.