What steps to take when trying to resolve unresponsive/hung/broken IIS web site?

Question

What steps do you take when you find an IIS web site is not responding?

I might try to first telnet to the specified port, then check web site binding and authentication, and at last restart it.

I think knowing what an experienced admin would check when facing such problems is quite useful.

In fact I my self spent over half an hour trying to figure out what is the problem and nothing seemed incorrect. I simply restarted the web site and problem was still there, but after restarting IIS service the problem was resolved.

If I could know a better tracing or at least a useful logging feature which helped me resolve it faster, that would save me over half an hour.

{FYI I am using IIS 7.5}

Answer 1

Determine Symptoms

Try to establish (as quickly as possible) the surface area of the problem:

• Connectivity? (Telnet is good; if you get an error page returned in the browser, something's obviously working - eliminate connectivity first)

• General App Pool failure, or specific to a content type? (Do ASPX files work/not work, but .HTM work? Do you have canary files for each app and content type?)

• Specific in-app failure, hang, or crash? (Most of this is for hangs and app failures; crashes dictate their own methodology: get a crash dump, debug it)

Collect Data

1. Grab whatever time-sensitive/timely data you will need to resolve the issue later. Don't worry about persistent stuff - Event Logs and IIS logs stick around, unless you're a compulsive clearer, in which case: stop it. (Those that don't have an Event Log of last week are doomed to repeat it)

2. Determine the affected worker process

   • APPCMD LIST WP can help with this, or the Worker Processes GUI at the Server level.

   • If using the GUI, don't forget to look at the Current Requests by right-clicking the worker process - that'll show you which module the requests are jammed in.

   • Determine the scope (just one App Pool, multiple App Pools, two with dependencies - this depends on your app and website layout)

   • Grab a memory dump of the worker process - once you've identified which App Pool has the problem, identify the relevant Worker Process, and use Task manager to create a memory dump by right-clicking that process. Note the filename for later.

   • On Task Manager: You need to use the same bitness of Task Manager as the Worker Process you're attacking with it - if you dump a 32-bit WP (w3wp*32) with 64-bit Task Manager, it's not going to be interpretable. If dumping a 32-bit process on 64-bit Windows, you need to exit Task Manager, run %WINDIR%\SYSWOW64\TaskMgr.exe to get the 32-bit version, then dump with the same bitness. (a ten second detour, but you must do it at the time).

Restore Service

1. Recycle the minimum number of Worker Processes in order to restore service.

   • Don't bother stopping and starting Websites, you generally need the App Pool to be refreshed in order to get the site working again, and that's what a Recycle does.

   • Recycling the App Pool is 9/10 times enough.

   • Note that recycling appears to happen on the next request to come in (even though the existing WP has been told to go away), so a worker process may not immediately reappear. That doesn't mean it hasn't worked, just that no requests are waiting.

   • IISReset is usually a tool used by people that don't know better. Don't use it unless you need every website to terminate and restart all at once. (It's like trying to hammer a nail into a wall with a brick. It might work, but you look like an idiot, and there's going to be collateral damage)

Determine Cause i.e. look at and think about the data you've collected.

1. Take the logs and the memory dump, look for commonalities, engage the app developers, debug the dump with DebugDiag 1.2, and so on.

Set up for next time

1. Don't assume it's the last occurrence - develop a plan for what you'll need to collect next time, based on this time.

   • For example, if the requests are all for the same URL, implement some additional instrumentation or logging, or a Failed Request Tracing rule that'll help identify the spot on the page that experiences a problem.

   • Performance monitor logs are helpful (if in doubt, get a perfmon log too).

Answer 2

Why would the bindings or authentication methods (which should be static) cause a site to be unresponsive? Those wouldn't be on my list of checks, or at the very least they wouldn't be on the top of my list.

The first thing I would check would be whether or not the site loads from the server itself. If it doesn't, you can rule out almost every possible network or DNS problem as the cause.