Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

For the last two weeks we are having intermittent severe spikes in system cpu usage (shown as %sys), which last for maybe half a minute, locking most processes, including ssh.

I've been trying to figure this out, but atop doesn't show anything relevant (system usage for processes it shows is insignificant), spikes are intermittent and I could not reproduce the spike using any workload for the web application this webserver hosts.

If you have any ideas on how to debug high %sys and (sometimes) %si CPU usage, please share them.

System specs (don't know if any of this is relevant): Dedicated server, CentOS 6, core i7 950, consistent 4 to 8 GB RAM free at any time, hard drives are in RAID-1.

Additional info:

  • dmesg output doesn't change between spikes
  • /var/log/messages doesn't change between spikes
  • Here is cat /proc/vmstat
  • Here is output of mpstat 1 during a typical spike

Add 07.11.11: looks like simple reboot restored system state, and we might never know what caused the disturbance in first place.

share|improve this question
    
You could put some files onto a webpage from the time where you see that high load or locks: screenshot of top, dmesg and/or /var/log/syslog, /proc/vmstat. You could remove sensitive data before if needed. –  ott-- Nov 3 '11 at 17:45
    
@ott-- added more info to the first post. –  Mark Nov 3 '11 at 19:20
    
Apparently, I can't add any more links to the post without being considered a spammer, here is output of iostat -x 5 during a typical spike. –  Mark Nov 3 '11 at 19:31
    
Do you run a BTRFS filesystem on Linux 3.0? –  mailq Nov 3 '11 at 22:01
    
@mailq: No, Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux, file systems are all ext3. –  Mark Nov 4 '11 at 5:18

5 Answers 5

I know this thread is a really old and I know you are already aware of this, %sys --> if the cycle is spent in %system then much of the execution is done in lower level code i.e might be issue on kernel side.If this issue is reproducible again,please collect the output of

echo t > /proc/sysrq-trigger

and attach /var/log/messages along with this thread.

share|improve this answer

High %si would suggest a high interrupt rate (si is the time spent in softirq handlers, AFAIK). Therefore my first guess would be that the server network interface is being hammered.

share|improve this answer
    
How can I prove or disprove this hypothesis? It does not seem like so, but it might be. –  Mark Nov 3 '11 at 19:25
    
That %si corresponds to time spent in softirq handlers? Well, if you don't believe me or some other documentation you might find, you can read the kernel source code. –  janneb Nov 4 '11 at 7:04
    
Err, no, hypothesis about network interface being hammered. –  Mark Nov 4 '11 at 7:09
    
Ah, you can check e.g. /proc/softirq; unfortunately I don't know of any tool that displays individual softirqs over time. Alternatively, run something like "dstat -ar --socket --tcp" when you get a spike and post the results. –  janneb Nov 4 '11 at 8:33
    
thanks a lot for dstat tool, it's kinda cool. Unfortunately, it's still not obvious, what causes spikes, see it for yourself: pastie.org/pastes/2821888/text?key=yqtv1iulh9nyhgahod1eq –  Mark Nov 6 '11 at 19:58
up vote 0 down vote accepted

It sounds stupid, but reboot helped and we might never know, what caused the spikes in first place.

Thank you for responses, though.

share|improve this answer
    
for the next one sar -I XALL 1 | grep -v 0.00 will tell you which interrups are getting that soft system time –  theist Jun 26 '13 at 13:47

On centos 6.2 and 6.3, disable huge page support:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
share|improve this answer
    
It would be helpful to explain what this does and why it would fix the problem. –  Michael Hampton Dec 2 '12 at 21:25

There are many factors contribute to the high %sys usage such as Logon, system call, context switch(both thread and procedure),IO and even sockets data copying from kernel mode to user mode. I suggest you can use sar, vmstat and iostat to check these out. Further more, it would be nice to find out which process caused the high %sys usage when spike. gdb would be helpful in this case.Find out the process and use gdb to attach on it and you will see what is going on with this process on this moment.Only thing you need to notice is this operation require debug information embedded in the procedure.

share|improve this answer

protected by Sven 2 days ago

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.