3

We have a memory leak plaguing many of our 2008R1SP2 file servers. The "File" non paged memory tag in poolmon.exe will leak until shared folders are no longer available, and then RDP goes down with it (and we have to reboot via console). Sometimes this leak is >1.5G/day.

poolmon

RamMap

If something needs to be removed, I need to have a pretty good hunch and a huge CM needs to be approved. The only change since this started happening was the addition of McAfee Agent 5.0.2.132, however AV with McAfee has not been running in any way (it's in preparation for a migration not yet implemented). I realize that the MFEm tag in the poolmon screen is McAfee, but it is not high in the list in most of the leaking servers.

Also, tags traced back to the NIC driver (either iANS or BCM8) are usually featured in the top 5-6 in the non paged bytes sort. We have not yet tried isolating from the network, but maybe this is a good idea.

The next logical step I understand is to run xperf / Windows Performance Toolkit, but this appears to require .NET 4.5, which we cannot install for a number of reasons.

1) Does anyone know of anywhere to download an older version of xperf that is standalone, compatible with 2008R1SP1 (even CLI only)?

2) Assuming #1 is unavailable (or at best a lousy idea), is there any other utility that can track and trace pool tags in a similar way?

3) Do you have any suggestions to point me in the right direction?

  • I posted a link to the MSI installer of xperf if you only want to install the WPT. Open the ETL on a second PC which has .net 4.5 installed. – magicandre1981 Oct 23 '15 at 4:01
1

It's maybe not a direct answer, but as I seen your "metafile" really high there is a private fix for that.

You experience performance issues in applications and services when the system file cache consumes most of the physical RAM - https://support.microsoft.com/en-us/kb/976618

Explained there: http://blogs.technet.com/b/mspfe/archive/2012/12/06/lots-of-ram-but-no-available-memory.aspx

1

We wound up paying for a support case with Microsoft. The engineer was able to trace the leak to the McAfee agent. Installing the following hot fixes (per Microsoft) resolved the non-paged pool leak, without having to uninstall the McAfee agent, on our 2008 SP2 servers: KB2029048, KB2961072, KB3019168. I hope this helps someone else who has this issue.

Thanks to all of those here who took the time to be helpful.

EDIT: In case you were curious, we could not disable the McAfee agent during troubleshooting due to political reasons. Multiple IT departments within a large company, with overlapping interests and some delicate egos. Now, the evidence speaks for itself at least. ;-)

  • thanks for posting the solution. How did they tracked it down? Via dump and my for_each command? – magicandre1981 Dec 26 '15 at 9:38
0

A high Meta cache + "File" usage indicates that programs do a lot of FileIO operations. You have use xperf to trace what causes the usage. Install the WPT from the Windows SDK (the 8.1 version also works on Win7/2008R2, but not the older Server 2008) (Here is the MSI installer if you don't want to install the .Net Framework 4.5), open a cmd.exe as admin and run this:

xperf -on PROC_THREAD+LOADER+POOL -stackwalk PoolAlloc+PoolFree+PoolAllocSession+PoolFreeSession -BufferSize 2048 -MaxFile 2048 -FileMode Circular && timeout -1 && xperf -d C:\poolusage.etl

capture 2-3 minutes of the pool usage grow. Open the ETL with WPA.exe (on a 2nd PC which has .net 4.5 installed), add the Pool graphs to the analysis pane.

Order the columns as you see them in the picture, load the symbols inside WPA.exe and expand the stack of the "File" tag that you saw in poolmon.

enter image description here

Here the File tag is used by locate32.exe which scans the file system to build in search index.

  • I set up a test environment and .NET 4.5 broke our services that we have running on these servers. These servers are all 2008R1SP2, so the 8.1 version of WPK is not compatible. The standalone MSI you linked installs, but I think it might actually be x32. It installs to Prog Files (x86). xperf.exe complains that it's not a valid Win32 application when run (under any creds). Is there any version of xperf/WPT that runs under Windows 2008 SP2 (non-R2) x64? – kiwisan Oct 23 '15 at 12:53
  • I was able to get a working version of xperf.exe by running the MS SDK bootstrap installer for Windows 7 on a x64 Win7 machine, then deselecting all but the redist for WPT and using the MSI it downloaded to C:\Program Files\Microsoft SDKs\Windows\v7.1\Redist\Windows Performance Toolkit. I'll update when I have more info from xperf. Thanks. – kiwisan Oct 23 '15 at 14:15
  • 1
    ok, if you need help. share the ETL, so that I can take a look at it – magicandre1981 Oct 23 '15 at 15:27
  • I ran your xperf code above, but I cannot get ANY pool data to show up in WPA (like in the screen you attached). I've tried the newest version of WPA (6.3.9600 on my Win10 machine, as well as the installed version of WPA on that server (4.8.7701). I have tried many different suggested CLI examples, and can't even get the memory category to show up. xperf version I think I'm limited to using is 4.8.7701. Here's the link to the etl generated from the xperf syntax above. – kiwisan Oct 24 '15 at 22:15
  • ok, You use the old 2008 Server with Sp1 (6.0), yes here the 8.1 version doesn't work. Run xperf -providers k and look if you see POOL in the result – magicandre1981 Oct 25 '15 at 6:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.