I have a set of very beefy, very underutilized servers which are running a couple virtual machines with Windows Server 2008 R2 and IIS 7.5.
The problem: Sometimes requests take a very long time to be processed. The user sees their browser spinning, apparently not getting any response from IIS.
Some stats & attempts at resolving:
- The load on the servers (host and VM) is negligible. CPU never goes over 5%, there are 10+ GB of RAM available, everything is connected via MPIO to a fast SAN.
- I get between 30 and 50 requests per second, a mix of dynamic and static content, only GET and POST. Most hits are cached (80% hit rate), so the load on the fabric/SAN/IO is close to nothing.
- TCP offloading is disabled on the host and virtual machine, both on the network adapter and via disabling TCP chimney
- The web applications are running in ASP.NET 4.0 Integrated Mode. They do not make long-running calls to third party web services or the like.
- I have tried both the processModel autoConfig, as well as setting maxWorkerThreads, maxConnections, maxIOThreads etc. to very high numbers with no difference.
- Database querys are all completing in less than 1 second. I profiled for an entire day and didn't capture a single query that would take longer.
- I've looked at tons of performance monitor counters; the ASP.NET queues and Application queues are always empty, nothing appears to get queued (which wouldn't make sense anyways considering that the server isn't breaking a sweat at all)
- I've identified using appcmd list requests that sometimes requests get 'stuck' for 20-60 seconds in IIS' "SendResponse" stage. This is the only thing I could find so far that would make any sense as to why we're seeing users getting stuck when navigating around. Note that most requests are getting processed fast though, it looks just like random requests from different application pools get stuck here.
Any idea what else I can look at? What would cause requests getting stuck in the 'SendResponse' stage in IIS?