Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This question already has an answer here:

I have a problem.

I am on a Xen VPN, running CentOS 6 and Apache. I have a medium traffic Zen Cart store and a few low traffic wordpresses. I have one or two very simple PHP form submission scripts, they are properly escaped.

Sometimes, my CPU usage will jump to 100% (actually 200%). This totally crashes the server - I can't login, can't SSH, can't access control panel (interworx), can't do anything. My only recourse is to reboot the server.

When I reboot, it is fine again...for a couple weeks.

If I don't reboot, the CPU will stay plateaued at 200% until I check and reboot. This went on for 24h once. When it does this, the network utilization drops to zero - this rogue process totally murders the whole server.

Because I can't ssh in, I can't use top or any direct tools like that.

Suspecting it is malware, I have combed through all the access logs each time this has happened. Yet, (I'm pretty sure) I can't find anything too suspicious (I get attacked a lot, but they all seem to be rejected). I have done my best to search for any backdoors or scripts that don't seem like they should be there.

My Zen Cart and Wordpresses are up to date. I use very secure passwords and only connect from linux terminals.

How can I investigate this? Are there any tools I can set that will somehow make a log of what the hell is happening when this happens? Is it more likely, from your gut intuition, to be an innocent software bug, or have my servers been compromised somehow and are being used for...I don't know what? Rogue researchers at Folding@Home?!? I can't think what hackers would be doing with maxed out CPU but zero network.

Any insight would be greatly appreciated. This is getting to be a big problem. Thank you!

share|improve this question

marked as duplicate by Chris S Feb 14 '13 at 1:39

This question was marked as an exact duplicate of an existing question.

Set up a cron job to run something like top -b -n1 every minute and dump the output to a file. This will give you a record of (a) what processes are consuming CPU and (b) the time processes are spending in different states. This may suggest where to concentrate your investigation.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.