Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an odd one that I just noticed yesterday. Over a 24 hour period, the cpu load on our apache machines (16 dual hex core machines) doubles. We handle about 8 billion requests per month, and everything sits behind haproxy load balancers. The number of running processes on each machine is always the same (capped in the load balancer), and any queue builds in haproxy. We limit the number of running sessions in haproxy to 90 per machine (chosen because of ram and cpu limits on each machine)

We're currently using the event mpm with the following config:

event MPM
<IfModule event.c>
    ServerLimit           15
    MaxClients           960
    StartServers           2
    ThreadsPerChild       64
    ThreadLimit           64
    MinSpareThreads       32
    MaxSpareThreads      112
    MaxRequestsPerChild 20000
</IfModule>

At the current rate, 20K requests should mean the thread is restarted every 2 minutes or so.

Memory usage is low. We have 12gb of ram on each machine, and at peak they only use about 7gb. A restart of apache immediately drops the load in half, and it stays that way for several hours.

Any ideas or additional info needed?

share|improve this question

closed as too broad by mdpc, Falcon Momot, Ward, dawud, Stephane Sep 20 '13 at 9:03

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

    
I'm going to test using prefork since we don't use keepalives. –  Brian Lovett Sep 18 '13 at 15:06
    
I tested 16 different configurations this morning using prefork, worker, and event. The ONLY configuration out of the 16 that hasn't resulted in an increase in cpu load over the course of the day is this one:<IfModule prefork.c> StartServers 90 MinSpareServers 90 MaxSpareServers 90 ServerLimit 90 MaxClients 90 MaxRequestsPerChild 10000 </IfModule> It was also the config that served the most requests over the test period. –  Brian Lovett Sep 18 '13 at 18:18
    
After about 8 hours of this config running the cpu load crept back up to double the norm. A restart of each server brought the loads back down to normal range (even though the number of requests to each machine was identical. Why does the cpu use and load increase over time even though the workload is identical? –  Brian Lovett Sep 19 '13 at 14:39
    
I'm going to test different maxRequestsPerChild to see if that makes a difference. I will split test 10K, 1K, and unlimited. –  Brian Lovett Sep 19 '13 at 14:43
    
I finally resolved this issue. As you can see, it took over a month of trian and error. Turns out, it was APC. APC cache by default has 32 or 64mb of cache, but our application code base has grown beyond that. After several hours, the cache would fill up, empty itself, then cpu load would increase due to cache slam. I ended up changing over to the new opcache from zend, and set aside plenty of memory (1gb) so we don't run into this issue again. So far, issue completely solved. –  Brian Lovett Oct 28 '13 at 19:12

1 Answer 1

What i can suggest you is that you inscrease the haproxy limit of 90 clients.

As you memory is about 7G in 12Gb you could do more clients.

My recomendations are the following:

If your app can handles mpm_worker, it would be much better to use worker than prefork for RAM usages.

It seems that you server is pretty much busy all the time, so make sanse Use a High start server, because apache will only start serving after create those process,so this will make your balance only start sending request to one server, when it is has lots of process to server ( this will cause i slow startup apache server)

I can try say more if i have more information... I mean, maybe you prefer having 6 machines with 12GB of ram.. when you could run in 4 machines with the right configuration!

I don't see why decreasing the MaxRequestPerChild wold do better on CPU, it only do better if you application misbehave. You have to use more CPU cycles to kill a process and start a new one. I mean, if you cut by half the MaxRequestPerChild it would have to create twice as more process than before.

CPU high doest mean a bad thing... It could maybe be doing a lots of work

What kind of application you run? Wich apache module... It seens that your application uses much more RAM than apache itself.

share|improve this answer
    
The issue isn't really resource usage. Take today for example. When I can in, the load on each server was over 15 with each server using 70% of the total cpu. After an apache restart with the exact same traffic volume, they have been running with a load average of 4 and 20-25% cpu load. Why does the cpu usage increase over time even though apache is serving the exact same number of requests? This happens over the course of 24 hours. –  Brian Lovett Sep 19 '13 at 14:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.