9

We have approximately 200 servers, Hyper V, File Cluster, and IIS, that are all experiencing the same issue, an event occurs on the server through normal use that maxes out or near maxes out the RAM on the server. Once this happens, the SVCHOST/Workstation service, specifically (weeded out by isolating the Workstation service to it's own SVCHOST) stops releasing handles/threads and the memory used by that service is never released. We have, in some extreme cases, Workstation services that are using as much as 40GB of ram on a 255GB server. Also finding upwards of 40 million handles in some cases.

On reboot, the problem of course, goes away, and doesn't appear again until all the memory has been used, say by the W3 process or the HyperV VMs, after that, the Workstation service starts grabbing all the RAM. The process is very slow and can take weeks/months depending on the amount of RAM on a server.

Both our Hyper V servers and IIS servers access shares for working files, these shares are on SSD storage, so they are plenty performant. We've installed all the current patches but have not moved to R2 as we have a lot of tooling in place that will make this a significant step and cannot find any clear indication that this would be fixed in R2.

We have run ProcMon and other tools but on the most problematic servers those tools won't even run. On the others, the results they provide just show that there appears to indeed be a memory leak in that process.

Is there a way we can free up the memory from this process or avoid the bug all together? We don't want to have to reboot and we cannot restart the process once it's in an error state. The process becomes frozen.

We're trying to avoid doing regular reboots to 'fix' this issue, so any answers would be appreciated.

13
  • What is your question? Nov 24, 2014 at 17:51
  • Indeed we do, but it's ambiguous at best, just thousands/millions of threads opening. On the most problematic systems we can't even run those tools, they just crash the server.
    – Craig
    Nov 24, 2014 at 18:22
  • We want to figure out a good solution to solve the problem other than rebooting the box. We are unable to stop the services once this problem starts.
    – Craig
    Nov 24, 2014 at 18:23
  • Has KB 2811660 been installed? Are these systems running server manager? support.microsoft.com/kb/2793908
    – user255580
    Nov 24, 2014 at 21:15
  • Yes, this KB was installed some time ago. Also, this leak is specific to the Workstation service, that KB applies to the WMI service.
    – Craig
    Nov 24, 2014 at 21:22

3 Answers 3

1

I had an eerily similar issue where the svchost was destroying the server performance.

The solution: Turns out I had a full Event Log. I cleared it out and everything was back up and running like nothing ever happened.

(I also recommend changing the size of the event log from the default, see below)

To set maximum log size by using the Windows interface
- Start Event Viewer.
- In the console tree, navigate to and select the event log you want to manage.
- On the Action menu, click Properties .
- In Maximum log size (KB) , use the spinner control to set the value you want and click OK .

It sounds exactly like what is happening here, but ended up being a really easy fix. A restart would temporarily solve the issue, but as soon as anything tried writing to the log, everything would spiral out of hand and just kept eating up resources.

Hope this helps!

-1
>Is there a way we can free up the memory from this process ?

There is no way you can externally (properly) release allocated memory or handle resources w/o killing the offending app.

>or avoid the bug all together? 

You are experiencing a memory and resource leak. The only way you will solve the problem is finding the leak and either avoiding its trigger (if possible) or fixing the leak at source code level; In the last case you need Microsoft help for producing the patch, but it seems they expect you to tell them "exactly" where the problem really is.

You can try to find the culprit by pinpointing the memory/resource leak by using i.e. MS Application Verifier

7
  • The trigger is file shares, which we cannot avoid.
    – Craig
    Dec 1, 2014 at 17:23
  • if you cannot avoid the trigger then find the leak with "Application Verifier" and contact MS with that info.
    – Pat
    Dec 1, 2014 at 22:44
  • The applications, as there are multiple, are all Microsoft. We've already contacted them, we're looking for a quicker solution as they are stating it may take them weeks/months to sort this out.
    – Craig
    Dec 1, 2014 at 22:49
  • Considering MS will not really rush for solving this kind of thing on a non-current OS I do not think you will find a quicker solution. A different thing is if you tell them where the leak is located.
    – Pat
    Dec 1, 2014 at 22:58
  • We have an open case and have been working with them for a month. The leak is literally in the Workstation service.
    – Craig
    Dec 1, 2014 at 23:14
-1

Crearing RAM is easy but no solution.

I suggest Sysinternals RAMMAP or VMMAP for deeper investigation. With this tools you can better see what happens. very often its a metafile problem.

Since Server 2008 we have this issue with all terminal servers running out of memory with an unbelievable memory consumption over time when starting applications from share.

Our workaround is hosting that applications on a separate Terminal Server and frequently clearing memory consumption.

We do this with a self designed c++ command line application using
SetProcessWorkingSetSize() with SeDebugPrivilege on all processes

Its strongly recommended not to do something like this ;)

1
  • Why downvote? Its exact what was asked for! It is no pleasure to try to help here...
    – Magnus
    Jan 20, 2016 at 7:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .