Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an apache process that suddenly appears on a particular server. When it shows up it starts consuming memory at a very rapid rate, then moves on to all the swap. In all it consumes about 11GB (including swap) of memory and the server eventually becomes unresponsive. The load on the server is under 1 at all other times. The process runs as nobody and I am having a hard time tracking down the source. If i run an strace on the process and all it did was continuously dump out mprotect over and over again

If i run lsof -p <pid>,

I get this, but only sometimes:

httpd   19229 nobody  152u  IPv4             175050 crawl-66-249-67-216.googlebot.com:62336 (CLOSE_WAIT)
httpd   19229 nobody  153u  IPv4             179104 crawl-66-249-71-167.googlebot.com:58012 (ESTABLISHED)

As long as I catch it, I can kill the process and the server almost immediately stabilizes. I have on site on the server that is getting a few thousand hits a a day that I think might be the source, but I still can't find the exact reason.

Also, this is a cPanel server and I have upcp'd the server, rebuilt apache with easy apache, and rebuilt httpd.conf.

It is not spawing any related processes, meaning I can find any php, mysql, cgi, etc. processes that relate to this process. It's just a loner process that balloons fast and consumes ever last MB of memory.

This is on a XenServer 5.6 based VM. No other servers in the cluster are having this issue.

share|improve this question
    
2 quick questions: does it always appear at the same time or interval (cron related?), and who is it's parent process ID when it does launch? –  troyengel Jul 18 '10 at 14:57
    
no consistent time interval, i'm trying to catch it again so i can get the parent process –  jemmille Jul 19 '10 at 0:43

2 Answers 2

  1. If you have PHP as a module, you wouldn't find it as a seperate process.
  2. Any self respecting cgi (or PHP as a module if so) should be able to limit it's memory consumption and die with a fatal error if going over the limit.
  3. Apache itself can use RLimitMEM.

Although limiting memory of a cgi/apache process does not solve the actual problem, it can keep it in check so a request can actually finish & write the offending url to an error log.

share|improve this answer
    
PHP is not compiled as a module, PHP memory consumption is properly limited in php.ini, it completely ignored RLimitMEM when I added it to httpd.conf, –  jemmille Jul 19 '10 at 0:23

Is it really a apache process or is it only running under the apache user?

If it is really a apache process you could look for big modules like mod_perl or mod_python, which could cause memory leaks.

you could also try to disable unnecessary apache modules.

strace -p PID

shows you what the process is actually doing (syscalls) - maybe that could give you some hints.

pmap PID

could give you some info about the memory structure of the process.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.