I've got an application running on IIS 6 that's being hit by hundreds of remote clients doing all kinds of requests, some that are long running and some that are not.
On some deployments, one in particular at the moment, the ASP .NET Applications - Requests Executing counter just keeps climbing until it hits the limit (5000) and then the server stops accepting requests.
Here are some very interesting facts when diagnosing the problem:
Thread count on the w3p processes and the System process are both low (under 100)
CPU on entire system is low
Total number of TCP connections listed in netstat -a (in any state) is arround 600, which is expected because we have about 600 remote clients.
So why would the request executing in IIS be at 5000 when really nothing is happening and everything seems to start and terminate properly?
I'd like to add that these are async calls using the IHttpAsyncHandler pattern. I believe that these do not have a timeout.