2

We are running an Apache 2.2 server for a very large web site. Over the past few months we have been having some users reporting slow response times, while others (including our resources, both on the internal network and our home networks) do not see any degradation in performance.

After a ton of investigation, we finally found a "Deny from none" statement in our configuration that was causing reverse DNS lookups (which were timing out) that solved the bulk of our issues, but we still have some customers that we are seeing in the Apache logs (using %D in the log format) with request processing times of > 300s for images, css, javascript and other static content.

We've checked all Deny / Allow statements for reoccurrence of "none", as well as all other things we know of that would cause reverse DNS lookups (such as using "REMOTE_HOST" in rewrite rules, using %a instead of %h in our log format configuration) as well as verified that HostnameLookups is set to "Off".

As an aside, we've also validated that reverse DNS lookups for folks having this problem do not time out - so I'm fairly certain DNS is not an issue in this case.

I've run out of ideas. Are there any Apache configuration scenarios that someone can point me to that I might be missing that would cause request times for static content to take so long only for certain users?

Thank you in advance.

1
  • We have also validated that the server is scaled correctly. We have plenty of connections to serve this content - coupled with the fact that this scenario is only happening for specific users.
    – user32326
    Jan 20, 2010 at 13:50

1 Answer 1

1

When I find this sort of thing, I first check:

  1. DNS. Use a network dump like tcpdump or wireshark to check for this, not just eyeballing the configuration file. If you're certain this is not the issue,
  2. What do traceroute / pings look like for those users? Do they all have something in common on their end? I've seen a bad NAT box cause no end of grief. I've also seen traffic local to a user cause my site to appear slower than it did for others without loaded connections, yet they NOTICE mine being slow.
  3. Firewall / tunneling. Are they doing something silly like blocking all ICMP? Are they on a tunnel? If yes to both, then chances are it's PMTU discovery timing out in some strange way.

Note that 300s response times probably means Apache gave up on them, not that it was served. 5 minutes is a very long time for the server to wait, but it's even more insane for a client to wait so long.

You must log in to answer this question.