8

For the last two weeks we are having intermittent severe spikes in system cpu usage (shown as %sys), which last for maybe half a minute, locking most processes, including ssh.

I've been trying to figure this out, but atop doesn't show anything relevant (system usage for processes it shows is insignificant), spikes are intermittent and I could not reproduce the spike using any workload for the web application this webserver hosts.

If you have any ideas on how to debug high %sys and (sometimes) %si CPU usage, please share them.

System specs (don't know if any of this is relevant): Dedicated server, CentOS 6, core i7 950, consistent 4 to 8 GB RAM free at any time, hard drives are in RAID-1.

Additional info:

  • dmesg output doesn't change between spikes
  • /var/log/messages doesn't change between spikes
  • Here is cat /proc/vmstat
  • Here is output of mpstat 1 during a typical spike

Add 07.11.11: looks like simple reboot restored system state, and we might never know what caused the disturbance in first place.

Improve this question
6
  • You could put some files onto a webpage from the time where you see that high load or locks: screenshot of top, dmesg and/or /var/log/syslog, /proc/vmstat. You could remove sensitive data before if needed.
    – ott--
    Nov 3, 2011 at 17:45
  • @ott-- added more info to the first post.
    – Mark
    Nov 3, 2011 at 19:20
  • Apparently, I can't add any more links to the post without being considered a spammer, here is output of iostat -x 5 during a typical spike.
    – Mark
    Nov 3, 2011 at 19:31
  • Do you run a BTRFS filesystem on Linux 3.0?
    – mailq
    Nov 3, 2011 at 22:01
  • @mailq: No, Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux, file systems are all ext3.
    – Mark
    Nov 4, 2011 at 5:18

5 Answers 5

Reset to default
4

I know this thread is a really old and I know you are already aware of this, %sys --> if the cycle is spent in %system then much of the execution is done in lower level code i.e might be issue on kernel side. If this issue is reproducible again, please collect the output of:

echo t > /proc/sysrq-trigger

And check system messages (var/log/messages or /var/log/syslog) to see if any thread may be using a lot of system CPU time.

Improve this answer
1

It sounds stupid, but reboot helped and we might never know, what caused the spikes in first place.

Thank you for responses, though.

Improve this answer
1
  • 2
    for the next one sar -I XALL 1 | grep -v 0.00 will tell you which interrups are getting that soft system time
    – theist
    Jun 26, 2013 at 13:47
1

On centos 6.2 and 6.3, disable huge page support:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
Improve this answer
2
  • 9
    It would be helpful to explain what this does and why it would fix the problem.
    – Michael Hampton
    Dec 2, 2012 at 21:25
  • Any explanation of link to more info?
    – JohnMudd
    Nov 16, 2020 at 22:03
0

High %si would suggest a high interrupt rate (si is the time spent in softirq handlers, AFAIK). Therefore my first guess would be that the server network interface is being hammered.

Improve this answer
5
  • How can I prove or disprove this hypothesis? It does not seem like so, but it might be.
    – Mark
    Nov 3, 2011 at 19:25
  • That %si corresponds to time spent in softirq handlers? Well, if you don't believe me or some other documentation you might find, you can read the kernel source code.
    – janneb
    Nov 4, 2011 at 7:04
  • Err, no, hypothesis about network interface being hammered.
    – Mark
    Nov 4, 2011 at 7:09
  • 1
    Ah, you can check e.g. /proc/softirq; unfortunately I don't know of any tool that displays individual softirqs over time. Alternatively, run something like "dstat -ar --socket --tcp" when you get a spike and post the results.
    – janneb
    Nov 4, 2011 at 8:33
  • thanks a lot for dstat tool, it's kinda cool. Unfortunately, it's still not obvious, what causes spikes, see it for yourself: pastie.org/pastes/2821888/text?key=yqtv1iulh9nyhgahod1eq
    – Mark
    Nov 6, 2011 at 19:58
-1

There are many factors contribute to the high %sys usage such as Logon, system call, context switch(both thread and procedure),IO and even sockets data copying from kernel mode to user mode. I suggest you can use sar, vmstat and iostat to check these out. Further more, it would be nice to find out which process caused the high %sys usage when spike. gdb would be helpful in this case.Find out the process and use gdb to attach on it and you will see what is going on with this process on this moment.Only thing you need to notice is this operation require debug information embedded in the procedure.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .