After scratching my head trying to figure out why my site was responding so slowly even though the server resources are fine, I finally checked the Apache status and found:
78 requests/sec - 0.7 MB/second - 8.5 kB/request 256 requests currently being processed, 0 idle workers
It appears that my apache is literally maxed out with connections. Anyone trying to visit my site gets put on a "waiting list" until Apache is free again.
It seems I have two options.
A) Raise the max connections limit above 256. Although according to this article it's not so easy:
By default, the MaxClients parameter has a compiled in hard limit of 256. This can be changed by recompiling Apache however. Some distributions, or hosting companies raise this limit to a very high value, such as 512 or even 1024 in order to cope with large loads.
B) Locate scripts that are taking up too much time. This seems a lot more tricky to me, since most apache processes just appear and then disappear again. Also, my sites PHP scripts are optimized pretty well...and once again, server resources are fine:
Server load 2.69 (8 CPUs) Memory Used 25.33% (2,039,108 of 8,048,804) Swap Used 1.32% (54,156 of 4,095,992)
Which option (if either of them) should I chose and how should I do it?
More information: Server Version: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/22.214.171.12435
HTTP Conf: http://pastebin.com/yBeLt6mP
Parital Request sample: http://pastebin.com/vzUVDMPR
Toggle Text-Wrap if the paste-bins show up weird.