I have a site which is receiving relatively high traffic of around 250.000 pageviews a day. The PHP code is usually executing in under 0.02 seconds, but the actual response time can go up during peak times to several seconds (sometimes 10+ seconds). Apache is running up to 700-800 processes, but memory usage is still low (not even half of the 8 GBs), server load is usually also very low.
I have moved images and asset files over to Amazon S3 and this has helped A LOT. It is also important, that there is also virtually constant uploading of new images to the site (as people post new photos).
From these clues I am thinking this might be a disk or perhaps a network issue, but how and with what tools can I diagnose this?