Request Executing leak in IIS 6.0 till Server Busy

Question

I've got an application running on IIS 6 that's being hit by hundreds of remote clients doing all kinds of requests, some that are long running and some that are not.

On some deployments, one in particular at the moment, the ASP .NET Applications - Requests Executing counter just keeps climbing until it hits the limit (5000) and then the server stops accepting requests.

Here are some very interesting facts when diagnosing the problem:

1. Thread count on the w3p processes and the System process are both low (under 100)

2. CPU on entire system is low

3. Total number of TCP connections listed in netstat -a (in any state) is arround 600, which is expected because we have about 600 remote clients.

So why would the request executing in IIS be at 5000 when really nothing is happening and everything seems to start and terminate properly?

EDIT

I'd like to add that these are async calls using the IHttpAsyncHandler pattern. I believe that these do not have a timeout.

Answer 1

"On some deployments" - what does this mean?

Short answer: Because your worker threads are getting tied up and hanging, and not timing out.

If it's ASP.Net, threads in retail mode should time out after 120 seconds.

As they're not, either there's something very serious going wrong, or you're running in debug mode, i.e. with <compilation debug="true"> set in your config files somewhere (machine or web.config).

This should never be set on a production web server.

If that's not the case, grab DebugDiag 1.2, and use it to evaluate memory dumps of your application while it's leaking threads. It should be able to show you the thread stack for leaked executing requests, and hopefully that helps you understand and fix the problem.

Answer 2

You state:

doing all kinds of requests, some that are long running and some that are not.

Long running requests will eventually kill your server. If you have a lot of these then you'll end up with thread pool exhaustion and hitting the max number of requests permitted.

If these long running requests are I/O bound (but not CPU bound) such as calling other web services or waiting for SQL operations to complete, then consider converting such pages to Asynchronous Pages:

ASP.NET Asynchronous Pages and when to use them

Wicked Code: Asynchronous Pages in ASP.NET 2.0

Performing Asynchronous Work, or Tasks, in ASP.NET Applications

If you're using ASP.NET MVC then you can do the same thing:

Using an Asynchronous Controller in ASP.NET MVC