Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We have two IIS 7 servers that run in web garden. There's a copy of every site on each server. Files are synchronized using DFS (only source files). Images & big files are located on different server using share.

Atleast few time a week, we get error "Not enough storage available" when reading any file or processing a server code. IIS Server is unable to map path to a local disk (not just the one that synchronizes with DFS), there also seems to be a problem loading group policy permissions (we get various errors) and we can't open any share on any server on lan. There are also I/O errors when reading files from other custom applications. Disk speed and latency during this fall-out are both ok. If we try to copy a file or test a drive speed, usually works. Sometimes, we can't even run "my computer" or any other software (for example performance monitor). It can only start if we "run as administrator". There's more than enough storage available on every disk.

Most of the sites stop working (but not all). After server restart, everything works ok for a few days and then the problem reoccurs.

Servers are virtualized using VMWARE, there are 50 IIS pools (some are mem limited) and around 200 running sites on each server. Both servers have 16GB of RAM (only about 80% used). CPU is constantly between 30 - 60%.

We tried changing the IRPStackSize registry property to a higher number, but there were no changes. Problems still exists. We tried to disable antivirus (NOD) – no success.

Any hint? Maybe we are close to IIS limits (to many sites/pools)? Maybe 32bit OS and 16GB of RAM? Maybe there is some 'secret' registry settings... We don't think that everyday restart is long-term solution.

share|improve this question
    
Anything in event viewer? –  ETL Mar 10 at 23:44
    
Have you investigated vcenter for memory constraints? Are these 2 IIS vm's in a resource pool? Any signs of Memory Ballooning or Paging at the VM or Resource Pool level? Do the vm's have Memory Limits vs the default Unlimited? Are there any issues with the datastore itself where the vm's memory files reside? –  Greg Drake Mar 11 at 15:24
    
@GregDrake We opened ticket at MS. We sent event viewer logs to them. Reply was: nothing unusual – change IRPStackSize. Yes, we are using PAE. Next, we'll investigate vcenter... –  easwee Mar 12 at 9:21

4 Answers 4

up vote 3 down vote accepted
+50

The Not enough storage available error has nothing to do with disk storage. It is about internal memory.

This error is quite easy to reproduce. Simply select a bunch of large files (images or DLL's or whatever), right-click and use the open-with option to open them all with Notepad. This should eat up allot of internal memory. When the swap-file nears it's maximum capacity, random services will start logging Not enough storage available to the eventlog. Often they will stay in a faulted state until they get restarted.

I have come across this issue with my own services, and had to add some extra error handling to force the service process to exit when such an error occurred on a thread. That way the service would restart and recover in an automated manner.

IIS usually handles this situation quite well (compared to an average windows service).

In order to find the process that causes this (usually a memory leak) you should use the performance monitor mmc snap-in, and record the memory usage of running processes (be selective when deciding what to record, since these logfiles can become quite large). Another option would be to use xperf, which is actually more suited for debugging low-level drivers and so, but can be helpful in this scenario as well.

If it turns out to be the w3wc service that's eating up all memory, you could use a tool such as Microsoft's all-new Application Insights (or any other IIS monitoring tool) to figure out which application pool, web-application or website is responsible. Then you can restrict memory usage for that particular pool, or schedule it to recycle gracefully.

IIS can usually cycle a pool in such a way that no one will notice (depending on how the app handles sessions, if it is custom built you might lose sessions when all static variables are flushed). Often a workaround like this is good enough, and expensive fixing of the bug can be avoided (disclaimer: that's not "my style" of solving problems, but most stakeholders love it that way).

share|improve this answer
    
Exactly, the physical resources are all ok: cpu 30-60%, ram cca 80% all the time, disks space ok... Fyi, we are monitoring the servers all the time. Also updates and hotfixes was first what we did. Just yesterday issue happened again. What we tried was to restart "Application Host Helper Service" an IIS per-se and sites started to work. Now we don't know are we cured the cause or the result of the problem? Maybe this is the new clue? –  easwee Mar 14 at 8:32
    
You are monitoring the servers, but are you monitoring the services and application pools individually? The 80% you say, was that by the time you got to look what was going on, or was the monitoring history constantly 80% before the error occurred? If it was constantly 80%, then it is the service that logged the Not enough storage available error, which is misbehaving by attempting to allocate an extreme amount of memory in a short time. You should be able to find the original process where the error originated in the event log. From your description it sounds like it could be explorer.exe ? –  Louis Somers Mar 14 at 16:40
    
While we still didn't manage to solve the issue, we believe this approach is quite close to what we really need to do in order to resolve this issue. –  easwee Mar 17 at 8:39

32bit Operating system with 16Gb ram will effectively leave 12Gb's unused. The upgrade to 64bit OS should defenitly be considered.

Use of the PAE switch (Physical Adress Extension) could offer a temporary fix.

For info with regards to max memory which a OS can use along with the PAE switch info look at:

http://msdn.microsoft.com/en-us/library/windows/hardware/gg487503.aspx

Regards,

share|improve this answer
    
Or simply reducing the amount of RAM. Could be running out of Page Table Entries. –  mfinni Mar 11 at 15:37
2  
Not "considered", baseline. Everyone running a 32 bit OS in 2013 wihtout REALLY good reason (special hardware with driver issues) should reconsider. His career, possibly. –  TomTom Mar 11 at 15:45

Could be a memory leak? Possible with that number of IIS pools, depending on what they're doing.

Easy to monitor with POOLMON.EXE from Microsoft (Grab it either from the official 2003 Support Tools, or a copy of just the file from my own space here : www.dfdfdf.eu/poolmon.exe)

Powershell script I run hourly for the duration of 'reboot -> crash'.

$date = get-date -Format ddMMyy-hhmm

.\poolmon.exe -b -n Output-$date.log

#send-mailmessage -smtpserver 10.1.1.11 -to patrick@mail.co.uk -from poolmon@mail.co.uk -subject "Poolmon output $date" -attachments "Output-$date.log"

#Extract top offender

$TAG = (Get-Content Output-$date.log)[3]

$Code = $((($TAG -replace '\s+', ' ').split(" "))[1]) 
$Byte = $((($TAG -replace '\s+', ' ').split(" "))[6])

ECHO "$Code,$Byte" >>chart.csv

Chart.CSV will show you the top TAG and its bytes allocated value.

The Output-$date.log will show you everything.

If you have got a single offender at the top, that stays at the top and increases until crash you have a likely candidate for the cause.

Also look in Eventvwr under system for events from SRV mentioning 'insufficient resources'

share|improve this answer
    
We will try to setup this - the bad thing is only that it takes time for the error to happen - but it's a start. Thx. –  easwee Mar 13 at 13:09

First of all, that is a lot of sites to be running with that many IIS worker processes on a 32-bit operating system. As mentioned by other users, a 64-bit operating system would definitely alleviate performance issues.

Secondly, you are receiving errors regarding storage but did not provide any information regarding storage space. Have you checked the drives on all servers to ensure there is adequate storage? Have you checked your virtual memory settings and verified the drive that the page file is on has plenty of space? In VMWare how are the virtual hard drives stored, locally or on shared network storage?

Third, there is no operating system information but being that IIS 7 is running I'm assuming Server 2008. Have you checked to see if there are service packs and hotfixes available for the servers and read the release notes? There may be a fix in one of those.

I would also recommend running PerfMon on all of the servers and looking for long Physical Disk queues, CPU use, Paging File % use, and Memory page faults/sec. Also look at your VMWare monitoring and see if there are any issues with the datastores such as high latency.

This issue could be caused by a number of things including lost communication with storage, page files growing past the limits of the drive or a number of other things. If you could post the exact text of the event in the event viewer and some other diagnostic information that would help. Being that you are having problems with group policies loading and other issues with applications not loading without jumping through some hoops I wouldn't completely rule out a corrupted operating system either at this point either.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.