9

I have a Windows 2003 Server running a bit slow where the commit charge and page file usage tick up and up until I eventually reboot. To fix this I need to find out what's consuming all the memory, and this is where the big mystery is.

Current stats from the Task Manager:

Physical Memory (K): 2096400
Commit Charge (K):   5364848
Page file usage:     5.11 GB

Fine, so let's pull up SysInternals Process Explorer and check the working set size of everything running. Biggest culprit is a Tomcat instance using 121,980K WS, 481,284K VM Size. Nothing in there comes close to explaining the 5 GB commit charge.

Next step: SysInternals pslist: pslist -m, split up the output by column and calculate column sums for the 61 processes that are reported.

SUM (Working Set)  : 681,484 K
SUM (Private Bytes): 593,424 K

Am I fundamentally misunderstanding what the tools are reporting? I've always been under the impression that an OS would actually commit much less memory than the full amount of virtual memory mapped by a process, on the assumption that it won't actually ever use that much, and that looking at VM here is a red herring.

1
  • 1
    Update: Yes, this system is paging, and I'm frequently getting the "Running out of virtual memory" warning. Whenever I get the warning I check the memory stats and there's no process that seems to be particularly misbehaving, but the Memory, Committed Bytes stat is always over 5 GB.
    – jpdaigle
    Jun 30, 2009 at 17:33

6 Answers 6

1

JP, I believe your comment that this is an automated build / unit test server that is running less than stable code says it all. My guess is that this server needs to be rebooted frequently because of the unstable code running on it probably has memory leaks that are giving you the symptoms you are seeing. Memory leaks are not going to get listed on any of the process trees of running processes as they are memory that is grabbed by processes that no longer exist, and are still allocated, even though the process associated with them are long gone.

Get a frequent maintenance window to allow you to automagically reboot this machine, perhaps nightly, a couple times a week, or weekly depending upon the speed at which this behavior starts to exhibit performance issues.

We have Oracle Application Servers servers on Windoze a while back that needed to be rebooted twice daily. We kicked out the users at lunch time, and in the middle of the night every day. That was a few years back, with supposedly stable productions code from Oracle.

2

Sorry to just paste some links - but we've been using some techniques to find memory leaks, using logparser and powershell, so I'll share.

http://blogs.msdn.com/debuggingtoolbox/archive/2008/08/22/powershell-script-chart-and-statistics-from-top-20-objects-leaking.aspx

(The Never Doubt Thy Debugger site is excellent): http://blogs.msdn.com/carloc/archive/2007/09/14/something-you-need-to-know-before-start-debugging.aspx

I'll also provide a link to Mark Russinovich's excellent insight to Memory Usage on Windows:

http://blogs.technet.com/markrussinovich/search.aspx?q=memory&p=1

Short of opening actual Microsoft cases, the Performance Team site has a lot of good info, too: http://blogs.technet.com/askperf/Default.aspx?p=2

1

The priciple tool for helping to diagnose this is perfmon. The counters I'd look at (to begin with) are:

Memory, Committed Bytes this is a measure of the demand for virtual memory
Process, Working Set, _Total virtual memory in "active" use
Paging File, %pagefile in use
Memory, Pages/Sec (this is not an necessarily an indicator of low memory, although everyone seems to assume it does)
Memory, Pages Output/Sec how many virtual memory pages were written to the pagefile to free RAM page frames for other purposes each second

Memory, Cache Bytes
Memory, Pool Nonpaged Bytes
Memory, Pool Paged Bytes
Memory, System Code Total Bytes
Memory, System Driver Total Bytes

when this sum gets close to 2GB you'll have performance issues usually due to 32 bit architecture

Memory, Available MBytes- this measures how much RAM is available to satisfy demands for virtual memory (either new allocations, or for restoring a page from the pagefile).

See also http://members.shaw.ca/bsanders/WindowsGeneralWeb/RAMVirtualMemoryPageFileEtc.htm

1
  • Interesting link. On this server, I'm seeing total Process Working Set at 640MB, but "Memory, Committed Bytes" is 5.4GB. None of the other counters you mention is very high. I frequently get the "Running out of virtual memory" warning, which is why I'm investigating. Adding comment to the OP.
    – jpdaigle
    Jun 30, 2009 at 17:31
1

How long has that server been running since the last reboot? It's an unfortunate fact of life that Windows has memory leaks. i.e. Memory that is allocated does not always get released back to the system. The symptoms are exactly what you are seeing. If you don't already do it I would suggest rebooting at least once a week. Many data centers reboot Windows every day, as doing so greatly cuts down on support calls. Of course that's a lot easier to do when you use clusters because there's no down time.

0

It's said that there is a sysadmin somewhere that really understands Windows memory stats, but I've never met them.

However it's not usually too hard to find a misbehaving process. Try using Perfmon to look at the various process counters like handles or Page faults/sec.

JR

Further thoughts: are you sure you're actually low on memory. The key thing is whether your'e getting paging. If there's no paging you're not short on memory. Using PerfMon have a look at the Memory - Pages/sec counter.

2
  • I don't see anything obvious in the running processes. Is there any possible way for memory to not be reclaimed by the OS after a process exits? This is an automated build/unit test server so of course processes crash all the time, as it's always running bleeding-edge builds automatically built from SVN.
    – jpdaigle
    Jun 30, 2009 at 15:12
  • See my edit for some further thoughts Jun 30, 2009 at 15:59
0

Remember that working set is the bit of memory that is actually represented by REAL physical memory. you also have your virtual memory charge, and Windows does code page sharing, some chunks of MFC are going to be shared among lots and lots of processes without duplicating memory usage. Plus you have Windows incessant desire to keep the filesystem cache as large as possible (i've seen it balloon to 800MB on a 2.5GB laptop before).

Windows memory management is very robust. However once Windows starts paging, things get ugly, and you ALWAYS have to leave room for the filesystem cache - always always always. I routinely have to shutdown and restart Firefox when my 80+ tabs drives it's memory consumption up to 1.2gb/1.2+gb.

Windows also blurs the line a bit, between pages that are in the cache, and pages that are listed as memory/working set.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .