12

I have an ASP.NET (v4.0) web app that is installed in a virtual directory (as an application) and is hosted in it's own app pool. This is repeated for each instance of the app (i.e. per customer).

The app pools are integrated (not classic) mode and LoadUserProfile is set to true. Otherwise, default settings.

Each instance currently has it's own copy of the code/config, and it's own data folder (basic file read/writes).

1 instance of this app runs well (operation used for comparison takes ~4 seconds). Every other instance runs slowly (from 10-25 seconds for the same operation).

If I move the slower instance to the "fastest" app pool that instance springs to life. If I move the faster instance into the slower app pool that instance slows to a crawl.

The app pools were created in the same way initially - manually. I later used the powershell copy routine to ensure an exact copy of the faster app pool and still the same behaviour. Comparing the apppool.config files shows they are identical barring the virtual directory assignments.

There are no shared resources that are being blocked, so far as I can tell, and I tested that by shutting down the performant app pool and restarting... slow is still slow, and then when I restart that app pool (so it's loaded last) it's still faster...

14
  • And the app folders are byte-identical?
    – usr
    Aug 11, 2012 at 19:06
  • Yes, just triple checked but the only difference is in the web.config where it specifies the name of the virtual directory it is being hosted under (and I double checked that was the only difference too) Every other file is byte-identical...
    – Ben
    Aug 12, 2012 at 22:34
  • What is the app doing that is taking longer in one apppool? Can you attach VS and pause the debugger to profile it?
    – usr
    Aug 12, 2012 at 22:42
  • I don't have VS installed on the server since it's a production system, but that looks like it's my next stop. I'm part way through adding massively verbose logging to the data access and file access components as the tracing we have has shown nothing specific as yet. I'll get some more stats and add it ASAP - I was optimistic that someone may have encountered similar so I could to avoid that path
    – Ben
    Aug 13, 2012 at 8:32
  • 1
    Since you are loading the user profile, that seems to be the key difference between those app pools. Check those users temp locations, file write permissions there, and if connecting to a database using the same user, then check permissions on the DB as well. All it takes is for a query to timeout for that user, so test with the same DB if possible. Good Luck!
    – Snives
    Aug 13, 2012 at 19:36

4 Answers 4

1

To further isolate the problem, I would suggest running Wireshark (or other packet analyzer of choice) on the host system, for two sessions. Assumption I'm making is that each app pool has either a unique IP assigned to it, or a unique port.

First get your baseline performance by filtering on the IP:port of your fast app pool. See what the traffic to and from the app looks like under "normal" conditions.

The second run, you will need to capture traffic from the slow/unresponsive app pool. If all network routing and such is correct up to the box, you should see repeated requests in one direction, most likely to the app from elsewhere, BUT if your app is something that makes a lot of requests to another server, your traffic may be heavy on egress instead of ingress.

This test will tell you if the problem is within the app, or if it's TCP/IP related issues that result in requests to the app timing out due to low/no communication.

Correlate the timestamps of your tests with the server's event logs and (if applicable) tracesink logs, and you should be able to zero in on the problem.

0

Failed Request Tracing (FRT) will be your best tool to track this down. It will show the pipeline and how long each step took to complete. That should point out whether it's something within the asp.net portion or if it's something within the IIS pipeline itself.

To setup FRT, from IIS at the site level create an FRT rule with an http status range of 200-999, and make sure to enable FRT (it's a separate step from the actions pane).

Then reproduce the issue and look at the generated files (%SystemDrive%\inetpub\logs\FailedReqLogFiles\w3svc{siteid}). Open them in Internet Explorer.

0

When IIS delays for long periods of time for no apparent reason it usually means that it is waiting for an outside service to time out. When it does then it tries another approach. For that matter this is true of just about anything in Windows or Linux. The first suspect for me in these situations is always network name resolution configuration. It's guilty until proven innocent.

Can this be recreated with one user hitting one of the duplicate sites? It would be good to know what the CPU and disks are doing when you recreate the 10 to 20 second response time. If it appears that not much is going on during the 10-20 seconds then you should check your binding names and name resolution for the bound names. If the CPU or disks are chewing away then you will need to figure out which process is working too hard and why.

Please post your findings, I am curious.

PS I would also check name resolution and access to any authentication services that may be in play. For example if the domain sever needs to be consulted make sure the first DNS server in your list is DNS server for the domain.

0

This is not the solution but we will work towards it;

  1. Run IISRESET (during maintenance hours) or kill the W3WP that belongs to the unresponsive app pool

  2. Launch the application

  3. While it is unresponsive, grab a dump file of W3WP that belongs to the slow app pool. Use either process explorer or task manager to create a dump file

  4. Grab the mscordacwks.dll and mscorwks.dll from under C:\Windows\Microsoft.NET\Framework64\v4.0.XXXXX

  5. Zip and upload these file somewhere from where I can download.

You must log in to answer this question.