I have a server running Apache/2.2.16 on Debian Lenny. The server is host to about 6 vhosts. One of the vhosts is just the FQDN for the server and doesn't have much of anything but an index.html page that Icinga checks every 5 minutes. Now, the server is generally under moderate load. It hosts a Joomla website with a Vbulletin forums section. It is also doing some ffmpeg stuff for the many videos on the site. It stays busy. Now, here is the kicker. Icinga is monitoring the FQDN for the server and not the main vhosts used to serve up content. Periodically Icinga will report HTTP critical, I'll log into icinga and run a "get /" by way of telnet or a "siege --get". Funny thing happens here, I don't get a response from Apache. No 200 code, nothing.
My first assumption is that Apache is jammed up so I go to pull up the other vhosts hosted on the server. They pull up fine but not without a slight delay. I scratch my head and then try to pull up the FQDN that Icinga is monitoring in my browser, it works too. Next I check the Apache logs for the FQDN vhost, I see the Icinga server hitting but minus a 200 error code. I view server-status and notice an average of about 60 request being processed with upwards of about 120.
I can't help but assume the issue is related to Apache and the limitations of suPHP. Eventually I would like to go with a modPHP/APC solution but for now suPHP is what I have to work with. Has anyone ever seen these sort of issues with Icinga monitoring? Does anyone have a recommended Apache mpm_prefork_module configuration? It is currently set to:
ServerLimit 512 StartServers 20 MinSpareServers 20 MaxSpareServers 40 MaxClients 512 MaxRequestsPerChild 0
Any insight or perspective is appreciated.
Also, the band-aid fix is to just cron an apache restart periodically, it works but I'm sure users get annoyed on the site.