High traffic, slow response: where is the bottleneck?

Question

I have a site which is receiving relatively high traffic of around 250.000 pageviews a day. The PHP code is usually executing in under 0.02 seconds, but the actual response time can go up during peak times to several seconds (sometimes 10+ seconds). Apache is running up to 700-800 processes, but memory usage is still low (not even half of the 8 GBs), server load is usually also very low.

I have moved images and asset files over to Amazon S3 and this has helped ALOT. It is also important, that there is also virtually constant uploading of new images to the site (as people post new photos).

From these clues I am thinking this might be a disk or perhaps a network issue, but how and with what tools can I diagnose this?

Thanks for the help.

Answer 1

There are many things it could be but start with the most likely things with an Apache/PHP server:

1. Memory -- If Apache is misconfigured it will eat up all the RAM and begin to use swap space which will kill your performance. Check to make sure your swap space is not used (or not used much). If it is reduce Apache's MaxClients or the memory consumption of whatever else is running on the server (MySQL, etc...). Since you say it is only using half then this may not be the problem, although 800 Apache processes in 4GB is only 5MB/process seems very low.
2. CPU -- PHP can be very CPU intensive, especially combined with other application/database. Check your server load and CPU usage in top to see how much you're using. Using a simple monitoring program like sar, or a more complex one like Nagios or Zabbix, can give you more detail on the CPU load over time. The one main thing you can do here is to install and use a PHP opcode cache like APC or eAccelerator. After that there is typically not much else you can do other than optimizing your database/application.
3. IO -- Check the output of top and iostat to see how much IO you are doing. A high wa% in top, either intermittent or constant, usually indicates an IO issue. If it is an IO issue then find out where it is occurring (Apache, PHP, database, something else, etc...) as the solution will ultimately depend on what is causing it.

I would also check the various log files for any obvious error messages (system, Apache, database, etc...). I know I've had sites with performance issues only because of something like the number of file handles was set too low for a high volume web site.