1

We've been having Out of Memory issues with our mixed classic ASP/.NET site for quite some time. Basically, we see a couple of different cases:

  1. Some request (SQL query that returns a LARGE recordset) happens that causes memory to drastically spike (Private and Virtual Bytes), and the IIS worker process crashes, and the application pool recycles.
  2. Private Bytes remain 'normal', but Virtual Bytes is at its max (~2GB), and the IIS worker process crashes, app pool recycles.

Crashes have the EventID's: 1009, 1011, & 1013.

In both cases, ASP reports Out of Memory messages in our application log.

If we truly are 'out of memory', why is that users ALREADY in the application can still work and new users are blocked from even logging in? Wouldn't that mean that NO requests could be served?

Obviously, IIS can still serve some requests -- so the answer is probably that it depends on the specific request. Maybe one is more 'expensive' than the other?

In any case, my ultimate question is -- at what point does the application pool say enough is enough, I MUST recycle?

We ran DebugDiag and saw some heaps that were over 90% or even 100% fragmented, so does the recycle happen only when fragmentation is high AND there is no more heap space?

This is a slightly subjective question but I hope I can get some input on sort of what that threshold is!

Thanks.

Improve this question

1 Answer 1

Reset to default
1

Unless the worker process breaks one of your configured recycling criteria (virtual bytes limit seems like a good one?) or is failing to respond to pings, it's not recycled.

Apps (ISAPIs) can report themselves as unhealthy, which triggers a recycle, but that happens under fairly narrow conditions.

Your app is basically fragmenting memory while it runs, and the OOM reflects this - no free contiguous, unallocated memory is available for new allocations; they fail.

Your options based on the description are:

  • fix the allocation pattern

  • separate the parts of the app into separate App Pools (run .Net separately from ASP, for example) - not all apps can be broken up in this way, but it's an easy win for those that can. Two App Pools = 2 (or more) worker processes = 2GB per worker process address space

  • implement a virtual bytes limit to recycle the app pool when it grows too big

  • implement a daily recycle to prevent the app getting to the point where its entire memory space is fragmented

  • if your app is stateless, try web gardening; increase max # worker processes in order to stave off failure for longer

  • Run as a 32-bit app on a 64-bit system; 4GB is provided to worker processes.

Essentially, if (whatever) heap's become super fragmented, you're in a death spiral at that point and you need a new worker process.

IIS doesn't have inbuilt safety belts for this, so implementing a recycling limit on your process before it gets to that point - or just saying "forget that, I'm going 64-bit!" might be possible solutions.

Improve this answer
2
  • Awesome answer. Your first few sentences helped wrap my mind around what was happening. I think our DebugDiag memory dump confirmed fragmentation, so we're investigating if we can slow that down on those heaps!
    – tresstylez
    Mar 5, 2012 at 22:03
  • Scaling out would reduce the fragmentation - if you can separate your (sub-)apps(?) into different App Pools, that'll buy more headroom for each, and fragmentation mitigation is all about headroom!
    – TristanK
    Mar 5, 2012 at 23:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .