Help trying to figure out why IIS7 is crashing / locking up / denying connections

Question

I've got a pretty busy website that is running on a single web front end machine, on W2K8 + IIS7.

Every now and then - eg. maybe monday at 3am or something, then a few days later .. some early morning time .. then nothing for 2 weeks ... etc - the website fails to respond to any client connections.

ie. no one can connect to the website.

I can remote desktop to the machine, etc no probs.

I restart the app pool (the website is running in intergrated mode), still nothing. I try and get a crash dump of the process (it's around 600 mb maybe even more) ... that fails after about a min of trying (and i have plenty of HD space).

The only way to fix this issue, is to manually stop the www service and then start it again. The stopping takes a while (a minute?) while starting is nearly instant.

I'm at a loss to figure out what part of my code is causing this. At first, I thought it might be a stack overflow because of some error that might be going to the error page, which in turn errors .. rinse repeat boom. But i've had a look at the error page and it feels ok.

So, I'm hoping someone might be able to help and say how I can correctly get a proper dump of the IIS process so i can then do some more autopsy on it. I would email Tess Ferrandez (the goddess of crash debuging) but I thought I'd try here before I spam her.

Can anyone have any suggestions to how I can figure out how to start to debug this issue?

Answer 1

Some things to consider:

1. When iis stops working, what is the www process doing? (ie consuming CPU, consuming memory, flat lined?)
2. From the server can you telnet to the website (hostname or ip address) or does the connection fail?
3. Does it always fail early in the morning? Are you running something around that time? Backups, batch processing, or database stuff?
4. Does your asp.net process consume a lot of ram or cpu?
5. If you create a second non-asp.net site (plain html), does it continue to function?

Answer 2

I've sort of found the answer a while back.

The application was crashing for whatever reason (i still couldn't figure out exactly where) BUT why IIS was stopping, was that it was happening a lot, in a short period -- IIS refers to this as RAPID-FAIL PROTECTION.

Here's a screen shot of some typical settings. If i understand this correctly, it's saying that if there's 5 crashes in 5 mins, then stop this application pool and reply with the Service Unavailable message. (In this screen shot, Rapid-Fail Protection is turned off, though).

_{(source: west-wind.com)}

So that's why my website was 'denying connections'. The app pool was told to STOP (collaberate and listen...) and returns a 503 for any new connections until a human (cough me cough) checked out what was happenining and manually restarted the app pool.