How is it possible that apache logs response time as significantly less than firebug?

Question

I am experiencing very slow page load times on the homepage of my wordpress blog. According to firebug, the average time spent waiting for the request to the homepage to return is 14s during peak hours.

To begin debugging this, I started logging the time it's taking for apache to serve the requests via the LogFormat directive. However, the average time that is being logged is around 1 second even though the requests are actually taking 14s. (this confused me)

I also am monitoring the RAM/CPU usage during peak hours, and there is plenty to spare.

I'm wondering if there are any problematic areas that I'm overlooking. I'm confused as to what could cause the request to the homepage take 14s when apache says it only takes 1s to render. (The 14s does not include any external page loads. Just the homepage http request).

I appreciate your thoughts and help!

Answer 1

Apache can't include time spent waiting for anything outside of the actual request processing, like TCP connection setup/teardown, network transit delays, load balancer queueing, proxy latency, and so on, whereas the Firebug time will be "SYN-to-FIN". Whilst 13 seconds seems excessive for the non-HTTP parts of the conversation, 1 second to generate a page is ridiculously long too, so it all seems badly out of whack to me. Rip out some tcpdumps of the whole packet stream for a request, that'll give you an idea of server-level network delays; if that doesn't account for it, start looking upstream of your server.