Httpd process consuming loads of memory

Question

I have a server which is giving me headaches. It hosts a couple of sites: these are either php based, or java based. I have a setup that uses apache2 + suPhp for the php sites, and apache2 + mod_proxy + apache tomcat for the java applications.

I've been seeing some strange behaviour over the past few weeks. Sometimes, I get a single httpd process that spikes to 30-40% cpu, and well over 70% memory. I don't see any php or java process taking up extra resources, so I (naively) assume that the problem is not related to the php or java code. These spikes seem to happen at random times and with random intervals; sometimes multiple times per day, other times nothing happens for a whole week.

Another strange thing I've noticed is that, when I manually kill -9 the httpd process that is acting up, another httpd process pops up within a few seconds and also starts taking up loads of memory and cpu. I can repeat this a few times, until it just stops happening all by itself :/

So, I actually have a few questions:

1. Does anyone have any tips on how I should trace the cause of this behaviour? Ultimately, I'd like to see what kind of request is causing these issues. I've looked at the httpd access and error logs, and I can't really find anything out of the ordinary. I'm not at all familiar with researching this type of issue, so even things that may seem very obvious to you could help.
2. Is there a way to limit the amount of resources a single httpd child process may consume? Or just plain kill it when it goes over a certain amount of memory?
3. Related to the previous question, could I perhaps configure the oom-daemon to be more trigger-happy on the httpd processes, and less on the other processes? I'm asking because, until I find a decent solution, I'd like to make sure it stops killing my java process when the httpd processes start acting up again.

---

UPDATE

Recently I found out that the request that is causing this mess comes from googlebot. This is an excerpt from the output of lsof for the process that is consuming all available mem and cpu:

httpd   18588 nobody   37u  IPv6 96675092               TCP myhost.com:http->crawl-66-249-76-96.googlebot.com:56730 (ESTABLISHED)

I set up mod_security to log all requests coming from the IP range that googlebots seem to use with this rule in my <VirtualHost>:

SecRule REMOTE_ADDR "@ipMatch 66.249.76.0/24" phase:1,id:1,auditlog,allow

I left the server like that for a while. During this time, the httpd process spiked several times, to an extent where the OOM daemon started killing processes (httpd, java, it even takes down mysql from time to time now). I then extracted all the urls that are hit by googlebot, and created a little script that curled all of those urls, hoping that I could make the httpd process spike and thus find the request that causes these issues.

Unfortunatly, that didn't happen - all the requests returned quickly, and the cpu and memory usage were nowhere near what they are when googlebot hits the server.

So I'm thinking there are 2 possibilities:

1. The problem is due to a specific HTTP header. My script doesn't replicate those, it just uses plain curl with no additional headers.

2. The request that causes the issues doesn't get logged. As far as I can tell, this shouldn't be the case since I'm telling mod_security to log the request in phase 1, which is before apache actually handles the request.

Does anyone have any other ideas?

---

UPDATE 2:

strace output of the process:

brk(0x3568c000)                         = 0x3568c000
brk(0x356ca000)                         = 0x356ca000
brk(0x35708000)                         = 0x35708000
brk(0x35746000)                         = 0x35746000
brk(0x35784000)                         = 0x35784000
brk(0x357c2000)                         = 0x357c2000
brk(0x35800000)                         = 0x35800000
brk(0x3583e000)                         = 0x3583e000
...
brk(0x3587c000)                         = 0x3587c000
brk(0x358ba000)                         = 0x358ba000
brk(0x358f8000)                         = 0x358f8000
brk(0x35936000)                         = 0x35936000
brk(0x35974000)                         = 0x35974000
brk(0x359b2000)                         = 0x359b2000
brk(0x359f0000)                         = 0x359f0000
brk(0x35a2e000)                         = 0x35a2e000
brk(0x35a6c000)                         = 0x35a6c000
brk(0x35aaa000)                         = 0x35aaa000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f2028000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f2005000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1fe2000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1fbf000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f9c000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f79000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f56000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f33000
...
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f10000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1eed000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1eca000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1ea7000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1e84000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1e61000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1e3e000
+++ killed by SIGKILL +++

Thanks in advance.

Answer 1

I don't see any php or java process taking up extra resources

That you think is worth checking at all, suggests you either know an awful lot about CGI vulnerabilities or very little about web serving / process management.

Ultimately, I'd like to see what kind of request is causing these issues

That's a sensible place to start. The easiest solution would be to install mod_security and configure it to log incoming requests (Apache only logs at the point where the response is dispatched). There are other approaches, such as sniffing the traffic (pastMon, Wireshark) or logging on a reverse proxy.

Is there a way to limit the amount of resources a single httpd child process may consume

Not directly, but you should set LimitInternalRecursion, LimitRequestBody, LimitRequestFields, LimitRequestFieldSize, LimitRequestLine, MaxKeepAliveRequests, MaxRequestsPerChild and Timeout to sensible values.

could I perhaps configure the oom-daemon to be more trigger-happy on the httpd processes

Messing the OOM Killer is nearly always a bad idea. Even if you think you know what you're doing. Does you java do anythiong useful in the absence of a webserver? If so then maybe you should think about running them on seperate machines.

Answer 2

First thing: you must limit the maximum number of apache processes that can be started. This can be done by lowering MaxSpareServers. Start low (3-10 and increase it alitle bit until the performance starts to go down). MinSpareServers should be 2.

PHP is usually running as an apache module (mod_php) this means that it runs in the same address space as apache, I mean you will see only apache process and inside will run php too. You can run gdb on these processes and run backtrace inside gdb to see what they are doing. You can use pstack for this too. If you notice for which URL is happening this (check access.log to find out the URL), then you can start debugging php code. Search for a php debugger or/and a php profiler.

Answer 3

Finally found the answer. Turns out one of the sites we host had a shady .htaccess hidden somewhere with a RewriteRule in it that triggers an endless loop in some cases.