Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got an application running on IIS 6 that's being hit by hundreds of remote clients doing all kinds of requests, some that are long running and some that are not.

On some deployments, one in particular at the moment, the ASP .NET Applications - Requests Executing counter just keeps climbing until it hits the limit (5000) and then the server stops accepting requests.

Here are some very interesting facts when diagnosing the problem:

  1. Thread count on the w3p processes and the System process are both low (under 100)

  2. CPU on entire system is low

  3. Total number of TCP connections listed in netstat -a (in any state) is arround 600, which is expected because we have about 600 remote clients.

So why would the request executing in IIS be at 5000 when really nothing is happening and everything seems to start and terminate properly?

EDIT

I'd like to add that these are async calls using the IHttpAsyncHandler pattern. I believe that these do not have a timeout.

share|improve this question
    
When you write "request executing", do you mean the request queue? –  Mathias R. Jessen Jan 20 '12 at 22:14
    
No i don't mean Request Queue. I mean the performance object is "ASP .NET Applications" and the counter is "Requests Executing" –  Mark Jan 20 '12 at 22:15
    
Just because the requests are being executed by a .NET application, doesn't mean that the server needs to do anything, they could be waiting for database transactions –  Mathias R. Jessen Jan 20 '12 at 22:23

2 Answers 2

"On some deployments" - what does this mean?

Short answer: Because your worker threads are getting tied up and hanging, and not timing out.

If it's ASP.Net, threads in retail mode should time out after 120 seconds.

As they're not, either there's something very serious going wrong, or you're running in debug mode, i.e. with <compilation debug="true"> set in your config files somewhere (machine or web.config).

This should never be set on a production web server.

If that's not the case, grab DebugDiag 1.2, and use it to evaluate memory dumps of your application while it's leaking threads. It should be able to show you the thread stack for leaked executing requests, and hopefully that helps you understand and fix the problem.

share|improve this answer
1  
DebugDiag + WinDbg + SOS = FTW! –  Kev Jan 22 '12 at 0:34
    
I just want to note that threads do not seem to be leaking. The thread count on the process is under 100 –  Mark Jan 24 '12 at 18:56
    
OK - take a look at a DebugDiag crash/hang report and see if you can see where the queue's forming. –  TristanK Jan 24 '12 at 21:10

You state:

doing all kinds of requests, some that are long running and some that are not.

Long running requests will eventually kill your server. If you have a lot of these then you'll end up with thread pool exhaustion and hitting the max number of requests permitted.

If these long running requests are I/O bound (but not CPU bound) such as calling other web services or waiting for SQL operations to complete, then consider converting such pages to Asynchronous Pages:

ASP.NET Asynchronous Pages and when to use them

Wicked Code: Asynchronous Pages in ASP.NET 2.0

Performing Asynchronous Work, or Tasks, in ASP.NET Applications

If you're using ASP.NET MVC then you can do the same thing:

Using an Asynchronous Controller in ASP.NET MVC

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.