For the last two weeks we are having intermittent severe spikes in system cpu usage (shown as %sys), which last for maybe half a minute, locking most processes, including ssh.
I've been trying to figure this out, but atop doesn't show anything relevant (system usage for processes it shows is insignificant), spikes are intermittent and I could not reproduce the spike using any workload for the web application this webserver hosts.
If you have any ideas on how to debug high %sys and (sometimes) %si CPU usage, please share them.
System specs (don't know if any of this is relevant): Dedicated server, CentOS 6, core i7 950, consistent 4 to 8 GB RAM free at any time, hard drives are in RAID-1.
- dmesg output doesn't change between spikes
- /var/log/messages doesn't change between spikes
- Here is cat /proc/vmstat
- Here is output of mpstat 1 during a typical spike
Add 07.11.11: looks like simple reboot restored system state, and we might never know what caused the disturbance in first place.