I have an apache process that suddenly appears on a particular server. When it shows up it starts consuming memory at a very rapid rate, then moves on to all the swap. In all it consumes about 11GB (including swap) of memory and the server eventually becomes unresponsive. The load on the server is under 1 at all other times. The process runs as nobody and I am having a hard time tracking down the source. If i run an strace on the process and all it did was continuously dump out mprotect over and over again
If i run
lsof -p <pid>,
I get this, but only sometimes:
httpd 19229 nobody 152u IPv4 175050 crawl-66-249-67-216.googlebot.com:62336 (CLOSE_WAIT) httpd 19229 nobody 153u IPv4 179104 crawl-66-249-71-167.googlebot.com:58012 (ESTABLISHED)
As long as I catch it, I can kill the process and the server almost immediately stabilizes. I have on site on the server that is getting a few thousand hits a a day that I think might be the source, but I still can't find the exact reason.
Also, this is a cPanel server and I have upcp'd the server, rebuilt apache with easy apache, and rebuilt httpd.conf.
It is not spawing any related processes, meaning I can find any php, mysql, cgi, etc. processes that relate to this process. It's just a loner process that balloons fast and consumes ever last MB of memory.
This is on a XenServer 5.6 based VM. No other servers in the cluster are having this issue.