Tracing and Correcting PHP Performance Problems

Question

I am running a small LAMP-based web server on which PHP-based pages seem to take a minimum of 5 seconds to render.

I believe that the problem is some issue with my PHP configuration in particular because:

• requests for static pages take 0.5s or less to satisfy
• requests for saved, static versions of PHP pages are served as fast as other static pages
• requests for PHP-based pages take >5s to satisfy whether or not they make requests to the DB
• the delay persists whether accessing the server by hostname or by IP, so it's not a DNS issue
• looking at vmstat during the request shows 0 for swap-in and swap-out, so it's not pagefile thrashing
• looking at top during the request shows that no apache process gets above 2-3% of CPU time, so it's not limited by CPU performance
• looking at the apache logs during requests shows a GET request for the PHP-based page, a 5-second delay, and then a request for the first CSS file that the PHP page loads.

On that basis, I feel pretty sure that the problem is with the PHP configuration, but PHP configuration is foreign to me. What are my options for improving PHP performance here? What are some common problems I should check for?

The main use case for this server is running a site based on the Joomla CMS, but this problem appears to be independent of Joomla, because the performance problems are happening with all PHP pages.

A clarification in response to Zoredache's question:

• On the server, running time curl http://127.0.0.1/foo/ is fast - tenths of a second.
• Elsewhere on the local network, running time curl http://10.1.0.1/foo/ is slow - 5 seconds at least.

Answer 1

Rasmus (Creator of PHP) did a whole talk on PHP performance http://talks.php.net/show/digg (There's a video of him doing the talk somewhere but I can't remember where, if I find it I'll update this post)

Edit;
The video is here