Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a server which is giving me headaches. It hosts a couple of sites: these are either php based, or java based. I have a setup that uses apache2 + suPhp for the php sites, and apache2 + mod_proxy + apache tomcat for the java applications.

I've been seeing some strange behaviour over the past few weeks. Sometimes, I get a single httpd process that spikes to 30-40% cpu, and well over 70% memory. I don't see any php or java process taking up extra resources, so I (naively) assume that the problem is not related to the php or java code. These spikes seem to happen at random times and with random intervals; sometimes multiple times per day, other times nothing happens for a whole week.

Another strange thing I've noticed is that, when I manually kill -9 the httpd process that is acting up, another httpd process pops up within a few seconds and also starts taking up loads of memory and cpu. I can repeat this a few times, until it just stops happening all by itself :/

So, I actually have a few questions:

  1. Does anyone have any tips on how I should trace the cause of this behaviour? Ultimately, I'd like to see what kind of request is causing these issues. I've looked at the httpd access and error logs, and I can't really find anything out of the ordinary. I'm not at all familiar with researching this type of issue, so even things that may seem very obvious to you could help.
  2. Is there a way to limit the amount of resources a single httpd child process may consume? Or just plain kill it when it goes over a certain amount of memory?
  3. Related to the previous question, could I perhaps configure the oom-daemon to be more trigger-happy on the httpd processes, and less on the other processes? I'm asking because, until I find a decent solution, I'd like to make sure it stops killing my java process when the httpd processes start acting up again.

UPDATE

Recently I found out that the request that is causing this mess comes from googlebot. This is an excerpt from the output of lsof for the process that is consuming all available mem and cpu:

httpd   18588 nobody   37u  IPv6 96675092               TCP myhost.com:http->crawl-66-249-76-96.googlebot.com:56730 (ESTABLISHED)

I set up mod_security to log all requests coming from the IP range that googlebots seem to use with this rule in my <VirtualHost>:

SecRule REMOTE_ADDR "@ipMatch 66.249.76.0/24" phase:1,id:1,auditlog,allow

I left the server like that for a while. During this time, the httpd process spiked several times, to an extent where the OOM daemon started killing processes (httpd, java, it even takes down mysql from time to time now). I then extracted all the urls that are hit by googlebot, and created a little script that curled all of those urls, hoping that I could make the httpd process spike and thus find the request that causes these issues.

Unfortunatly, that didn't happen - all the requests returned quickly, and the cpu and memory usage were nowhere near what they are when googlebot hits the server.

So I'm thinking there are 2 possibilities:

  1. The problem is due to a specific HTTP header. My script doesn't replicate those, it just uses plain curl with no additional headers.

  2. The request that causes the issues doesn't get logged. As far as I can tell, this shouldn't be the case since I'm telling mod_security to log the request in phase 1, which is before apache actually handles the request.

Does anyone have any other ideas?


UPDATE 2:

strace output of the process:

brk(0x3568c000)                         = 0x3568c000
brk(0x356ca000)                         = 0x356ca000
brk(0x35708000)                         = 0x35708000
brk(0x35746000)                         = 0x35746000
brk(0x35784000)                         = 0x35784000
brk(0x357c2000)                         = 0x357c2000
brk(0x35800000)                         = 0x35800000
brk(0x3583e000)                         = 0x3583e000
...
brk(0x3587c000)                         = 0x3587c000
brk(0x358ba000)                         = 0x358ba000
brk(0x358f8000)                         = 0x358f8000
brk(0x35936000)                         = 0x35936000
brk(0x35974000)                         = 0x35974000
brk(0x359b2000)                         = 0x359b2000
brk(0x359f0000)                         = 0x359f0000
brk(0x35a2e000)                         = 0x35a2e000
brk(0x35a6c000)                         = 0x35a6c000
brk(0x35aaa000)                         = 0x35aaa000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f2028000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f2005000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1fe2000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1fbf000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f9c000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f79000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f56000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f33000
...
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1f10000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1eed000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1eca000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1ea7000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1e84000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1e61000
mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f72f1e3e000
+++ killed by SIGKILL +++

Thanks in advance.

share|improve this question
    
Do you have any cron jobs that are hitting these services? –  HTTP500 Aug 7 '12 at 12:43
    
Well, there are other services which use the services hosted on the problematic machine on a regular basis, but those do not seem to be related to the issue. I've monitored cpu/mem usage several times during the execution of these services and the resource consumption did not even come near to what I'm seeing on those httpd processes. Also, when these services get used, it's either the php or java process that uses more resources, not the httpd process itself. On top of that, the issues seem to come at way to irregular times to be caused by a cron job. –  Mopper Aug 7 '12 at 13:09
add comment

3 Answers

I don't see any php or java process taking up extra resources

That you think is worth checking at all, suggests you either know an awful lot about CGI vulnerabilities or very little about web serving / process management.

Ultimately, I'd like to see what kind of request is causing these issues

That's a sensible place to start. The easiest solution would be to install mod_security and configure it to log incoming requests (Apache only logs at the point where the response is dispatched). There are other approaches, such as sniffing the traffic (pastMon, Wireshark) or logging on a reverse proxy.

Is there a way to limit the amount of resources a single httpd child process may consume

Not directly, but you should set LimitInternalRecursion, LimitRequestBody, LimitRequestFields, LimitRequestFieldSize, LimitRequestLine, MaxKeepAliveRequests, MaxRequestsPerChild and Timeout to sensible values.

could I perhaps configure the oom-daemon to be more trigger-happy on the httpd processes

Messing the OOM Killer is nearly always a bad idea. Even if you think you know what you're doing. Does you java do anythiong useful in the absence of a webserver? If so then maybe you should think about running them on seperate machines.

share|improve this answer
    
Thanks for the hints, I'll be trying those later on. We don't have any sysadmins here atm so I'm kind of on my own without any experience in configuring (production-use) servers. As for the java process being useful without apache2 in front of it: yes, it does handle more stuff than just requests coming from the web, + it takes a whole lot longer to restart a tomcat process than it does for an httpd process, which is why I'm eager to keep it alive. –  Mopper Aug 7 '12 at 14:25
add comment

First thing: you must limit the maximum number of apache processes that can be started. This can be done by lowering MaxSpareServers. Start low (3-10 and increase it alitle bit until the performance starts to go down). MinSpareServers should be 2.

PHP is usually running as an apache module (mod_php) this means that it runs in the same address space as apache, I mean you will see only apache process and inside will run php too. You can run gdb on these processes and run backtrace inside gdb to see what they are doing. You can use pstack for this too. If you notice for which URL is happening this (check access.log to find out the URL), then you can start debugging php code. Search for a php debugger or/and a php profiler.

share|improve this answer
add comment
up vote 0 down vote accepted

Finally found the answer. Turns out one of the sites we host had a shady .htaccess hidden somewhere with a RewriteRule in it that triggers an endless loop in some cases.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.