4

So I'm sure this has probably been asked before, but I want to make sure that I'm getting the right info.

I'm running a 360 Linode box with Debian 5 and Apache 2.2. I've compiled everything myself (no apt-get). Every now and then (once every few weeks?), my server would randomly "crash": it would shoot up to 100% CPU (really 400%, but you know what I mean) and hand the box. You couldn't SSH in to see what was the problem and the server itself stopped accepting connections. The only way to fix it was to reboot the box.

Recently, it's started happening with relative frequency: 24 hours, 12 hours, 10 hours, 8, 6, 4. Finally, I managed to catch a glimpse right before it locked up about two days ago. I noticed that the disk IO was elevated and that there was barely any RAM left! Also, there was a boat load of httpd processes running at 3-4% RAM. And by boat load, I mean when I did a ps -ef, they took up the entire screen. If you scrolled up, they took up the entire buffer for my SSH client.

So I made some tweaks to my code thinking that something wasn't closing properly. I fixed memory issues in my PHP, I turned on more granular error logging and fixed a bunch of errors, and it seemed to help to some degree. The crashes went back to about every 24 hours.

I'm convinced that this is caused because there is too little memory, and the volume of hits I receive is kicking my server into the swap. Since there are so many requests hitting the swap, the disk IO shoots through the roof, causing my CPU usage to shoot through the roof, causing my server to lock up.

Here's what I did to try to fix it: I did some research and found that I should probably be using prefork. I looked around in my config and couldn't find any ServerLimit or MaxClients or anything of the like, so I added some "default" values in and my server refused to accept /any/ incoming connections. Effectively, the prefork values blocked all inbound http traffic by choking off the connection (maybe because my server isn't capable of handling prefork? idk).

The way I see it, what my server "used to do" was fine, except all of those apache process kept hanging up and leaking memory. Is there any way to set a timeout on Apache processes or put a limit on how many of them there are? It seems pretty dumb that the best solution is to use prefork; I have to imagine that there's a better way.

Thanks guys

2
  • "I'm running a 360 Linode box with Debian 5 and Apache 2.2. I've compiled everything myself (no apt-get)." Why would you do this? This is a recipe for making your life way harder than it needs to be, and for removing any opportunity for getting upstream support. Try this again with the distro supplied packages, would be my recommendation.
    – Graeme
    Nov 6, 2009 at 19:38
  • @Greame: PHP 5.3 is not available for Debian via apt-get. It's easier to compile PHP and apache from source than to hack PHP 5.3 into the distro package. I'm looking for an answer to a question about mpm/Apache memory issues, not a suggestion to dump my entire architecture and start over.
    – mattbasta
    Nov 6, 2009 at 20:10

4 Answers 4

1

It seems you are learning the hard way that PHP is a memory hog and not particularly scalable.

Some suggestions, in no particular order:

If you still suspect a memory leak set MaxRequestsPerChild to really low value.

Consider buying more memory, 360 megs is really not much nowadays.

Try finding an average size of httpd process by running ps or top and then setting MaxClients so that everything always fits in memory. Swapping is a death spiral, the slower you are processing the requests because of it, the more processes apache needs to fork, using even more memory.

If you laod php as a module in apache it gets loaded for every request, be it a script or a static file (an image or css or .js or whatnot). Consider serving static content from a separate server or using fastcgi or a reverse proxy like nginx to have apache serve only php to limit the number of fat php instances you have to keep in memory.

2
  • Thanks for the advice. I bit the bullet and upgraded to a 540M box. What is your opinion on lighthttpd?
    – mattbasta
    Nov 8, 2009 at 3:01
  • lighttpd doesn't actually buy you all that much memory itself as much as it forces you to use FastCGI for php, which is usually (depending on settings) a more memory efficient configuration. You can just use FastCGI in apache via mod_fcgid or you can go with nginx's fastcgi setup.
    – cagenut
    Dec 7, 2009 at 15:54
0

I guess your machine is thrashing with low memory. In my experiance every apache process that is running has the potential for taking up loads of memory. If you have php running as a module in apache look in your php.ini what the value for memory_limit = is. I had 128M which is quite a lot when you have only 10 apache processes running. They may not take that amount from the beginning but if your PHP applicationleaks memory or really needs 128M you can hit your servers limit quite easyly.

My recommendation is: Physical Ram divided by memory_limit equals max_procs

1
  • I don't run anything memory "intensive", so I never reach anything close to the limit. Mine is set at 64M because I run a couple intense PHP CRON operations (once a day or so). The majority of the "idle" processes are around 11/12MB, though.
    – mattbasta
    Nov 8, 2009 at 3:03
0

If your really low on memory you could try running something like lighttpd or nginx with php as a fastcgi process, I haven't used nginx that much but I use lighttpd allot and it has an extremely low memory/cpu overhead.

Lighttpd and php via fastcgi tutorial

0

Do you have any database backend you are connecting to? Often what will happen is if some queries start slowing down, you'll have apache processes begin to back up and given the low amount of RAM on your linode, it will snowball and fall over causing the CPU and load to shoot up.

Another thing, you mentioned you are using the worker model. This is fine if you are 100% certain all the modules you are using within PHP are threadsafe. The actual php installation manual recommends against using it though ( php manual )... Regardless of which model you go with, you need to make sure you tune them properly. A good starting point is Tuning LAMP systems. If MaxClients isn't tuned properly Apache could end up causing your system to crash as it takes up all memory during a burst of traffic to your site.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .