We recently had some maintenance done on our development server; we don't know what that consisted of as we do not maintain our servers. Since then we've been experiencing a rash of HTTP requests that last for 20s+ before being completed, or, more frequently, just timeout. This is a huge issue for us, as we use an uncompiled RequireJS on the dev server, so any given page performs anywhere from 100 to 400 requests for files or data, and even if all of them complete successfully, the page itself may take minutes to load waiting for requests to finish. We initially thought it was some form of DDoS protection added in, so we shifted everything over to local Apache servers. However, due to the configuration differences, we're trying to get the dev server fixed and move everything back to there.
We started off trying to figure what exactly was causing the issue (Apache, PHP or an extension, MySQL, etc). Initially, the group that runs our servers claimed that it was most likely our code base, or possible the fact that we use the Zend framework as our base. While looking into this, the first obvious thing we noticed was that PHP requests never seemed to experience any huge issue. They may fluctuate in time, but never time out. The only requests that timeout are to static resources, such as CSS, LESS, JS, or HTML files. By extension, we've also ruled out any long MySQL requests
To further enforce this, we tried using
time from the PHP CLI, which results in expected execution times, save for at times it takes ~7s for the process to actually start. We've even duplicated this issue with a simple "Hello World" script that did not load Zend. We think that this long start-up may be because of the cURL extension (https://bugs.php.net/bug.php?id=50410), but are not sure.
By chance, when working on some IE9 compatibility issues, we discovered that while using IE9, we experienced absolutely no timeouts, which has been reproduced over several days while simultaneously having other browsers timeout. This was surprising, as we figured of any browsers to have issues, IE would be one of them. Even more surprising, we used extensions in Chrome and Firefox to change the request headers to IE9 headers, and the problem disappears completely!
Since we don't know what exactly is causing the problem, it's hard to get someone to actually take a in-depth look at the issue. If we can find something more for them to go on, we think that could speed up the entire process. We're currently in the process of attempting to get some debugging tools installed to help profile the server to see if there are any bottlenecks, but we don't know how long this will take. This has been going on for about 3 weeks now, and at this point were not sure what else to check, so any suggestions would be greatly appreciated.
So, to recap:
- Development Server recently started timing out. Staging and Live servers unaffected.
- The timeouts seem to only occur on static files (JS, CSS, LESS, HTML, etc)
- The timeouts almost never occur in Internet Explorer
- Changing the request headers in FF or Chrome to emulate IE stops the problem in those browsers as well
- Running simple scripts from PHP CLI will periodically cause a ~7s delay before beginning to execute the script.
I can try to provide extra information as needed, but it is very hard to get server configuration information from this group, they claim due to security issues.