My normal CPU load average is less than 1.0. However, once in a while for some unknown reasons, the CPU load average shoots up to 23.21, even though the connections and processes running are the same as the normal condition. I am unable to root cause the reason for high CPU load average. Here's the top output. `
top - 14:05:03 up 2 days, 3:04, 1 user, load average: 23.21, 22.71, 21.98
Tasks: 244 total, 3 running, 241 sleeping, 0 stopped, 0 zombie
Cpu0 : 17.6%us, 77.5%sy, 2.9%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.0%si, 0.0%st
Cpu1 : 21.6%us, 75.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.9%si, 0.0%st
Cpu2 : 16.7%us, 80.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.9%si, 0.0%st
Cpu3 : 17.6%us, 79.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.9%si, 0.0%st
Cpu4 : 17.6%us, 79.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.9%si, 0.0%st
Cpu5 : 27.5%us, 70.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.0%si, 0.0%st
Cpu6 : 17.6%us, 79.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.9%si, 0.0%st
Cpu7 : 17.0%us, 81.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.0%si, 0.0%st
Mem: 12301916k total, 11569724k used, 732192k free, 585756k buffers
Swap: 1052248k total, 8k used, 1052240k free, 8482092k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5635 apache 16 0 197m 20m 14m S 8.8 0.2 0:01.39 php-cgi
4160 dieselst 16 0 202m 25m 17m S 6.9 0.2 0:03.16 php-cgi
`
This load average will not go down, no matter what. This server serves the apache with php and mysql. So if I shutdown apache server and mysql server, the load still stays the same. How can I find out who's the culprit?
Thanks, Ketan