So I'm sure this has probably been asked before, but I want to make sure that I'm getting the right info.
I'm running a 360 Linode box with Debian 5 and Apache 2.2. I've compiled everything myself (no apt-get). Every now and then (once every few weeks?), my server would randomly "crash": it would shoot up to 100% CPU (really 400%, but you know what I mean) and hand the box. You couldn't SSH in to see what was the problem and the server itself stopped accepting connections. The only way to fix it was to reboot the box.
Recently, it's started happening with relative frequency: 24 hours, 12 hours, 10 hours, 8, 6, 4. Finally, I managed to catch a glimpse right before it locked up about two days ago. I noticed that the disk IO was elevated and that there was barely any RAM left! Also, there was a boat load of httpd processes running at 3-4% RAM. And by boat load, I mean when I did a ps -ef, they took up the entire screen. If you scrolled up, they took up the entire buffer for my SSH client.
So I made some tweaks to my code thinking that something wasn't closing properly. I fixed memory issues in my PHP, I turned on more granular error logging and fixed a bunch of errors, and it seemed to help to some degree. The crashes went back to about every 24 hours.
I'm convinced that this is caused because there is too little memory, and the volume of hits I receive is kicking my server into the swap. Since there are so many requests hitting the swap, the disk IO shoots through the roof, causing my CPU usage to shoot through the roof, causing my server to lock up.
Here's what I did to try to fix it: I did some research and found that I should probably be using prefork. I looked around in my config and couldn't find any ServerLimit or MaxClients or anything of the like, so I added some "default" values in and my server refused to accept /any/ incoming connections. Effectively, the prefork values blocked all inbound http traffic by choking off the connection (maybe because my server isn't capable of handling prefork? idk).
The way I see it, what my server "used to do" was fine, except all of those apache process kept hanging up and leaking memory. Is there any way to set a timeout on Apache processes or put a limit on how many of them there are? It seems pretty dumb that the best solution is to use prefork; I have to imagine that there's a better way.