Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

For the last two weeks we are having intermittent severe spikes in system cpu usage (shown as %sys), which last for maybe half a minute, locking most processes, including ssh.

I've been trying to figure this out, but atop doesn't show anything relevant (system usage for processes it shows is insignificant), spikes are intermittent and I could not reproduce the spike using any workload for the web application this webserver hosts.

If you have any ideas on how to debug high %sys and (sometimes) %si CPU usage, please share them.

System specs (don't know if any of this is relevant): Dedicated server, CentOS 6, core i7 950, consistent 4 to 8 GB RAM free at any time, hard drives are in RAID-1.

Additional info:

  • dmesg output doesn't change between spikes
  • /var/log/messages doesn't change between spikes
  • Here is cat /proc/vmstat
  • Here is output of mpstat 1 during a typical spike

Add 07.11.11: looks like simple reboot restored system state, and we might never know what caused the disturbance in first place.

share|improve this question
    
You could put some files onto a webpage from the time where you see that high load or locks: screenshot of top, dmesg and/or /var/log/syslog, /proc/vmstat. You could remove sensitive data before if needed. –  ott-- Nov 3 '11 at 17:45
    
@ott-- added more info to the first post. –  Mark Nov 3 '11 at 19:20
    
Apparently, I can't add any more links to the post without being considered a spammer, here is output of iostat -x 5 during a typical spike. –  Mark Nov 3 '11 at 19:31
    
Do you run a BTRFS filesystem on Linux 3.0? –  mailq Nov 3 '11 at 22:01
    
@mailq: No, Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux, file systems are all ext3. –  Mark Nov 4 '11 at 5:18
show 1 more comment

4 Answers 4

I know this thread is a really old and I know you are already aware of this, %sys --> if the cycle is spent in %system then much of the execution is done in lower level code i.e might be issue on kernel side.If this issue is reproducible again,please collect the output of

echo t > /proc/sysrq-trigger

and attach /var/log/messages along with this thread.

share|improve this answer
add comment

High %si would suggest a high interrupt rate (si is the time spent in softirq handlers, AFAIK). Therefore my first guess would be that the server network interface is being hammered.

share|improve this answer
    
How can I prove or disprove this hypothesis? It does not seem like so, but it might be. –  Mark Nov 3 '11 at 19:25
    
That %si corresponds to time spent in softirq handlers? Well, if you don't believe me or some other documentation you might find, you can read the kernel source code. –  janneb Nov 4 '11 at 7:04
    
Err, no, hypothesis about network interface being hammered. –  Mark Nov 4 '11 at 7:09
    
Ah, you can check e.g. /proc/softirq; unfortunately I don't know of any tool that displays individual softirqs over time. Alternatively, run something like "dstat -ar --socket --tcp" when you get a spike and post the results. –  janneb Nov 4 '11 at 8:33
    
thanks a lot for dstat tool, it's kinda cool. Unfortunately, it's still not obvious, what causes spikes, see it for yourself: pastie.org/pastes/2821888/text?key=yqtv1iulh9nyhgahod1eq –  Mark Nov 6 '11 at 19:58
add comment
up vote 0 down vote accepted

It sounds stupid, but reboot helped and we might never know, what caused the spikes in first place.

Thank you for responses, though.

share|improve this answer
    
for the next one sar -I XALL 1 | grep -v 0.00 will tell you which interrups are getting that soft system time –  theist Jun 26 '13 at 13:47
add comment

On centos 6.2 and 6.3, disable huge page support:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
share|improve this answer
    
It would be helpful to explain what this does and why it would fix the problem. –  Michael Hampton Dec 2 '12 at 21:25
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.