I have a custom server application that runs on Windows 2008 R2. It is a home grown Windows Service written in .Net supporting a number of custom terminals. I have a test machine that has a similar specification to the live server and I have a set of client simulators that I can use to produce a load that is a reasonable approximation of the real system. I need to be able to support 12,000 of these and at present the server is running out of memory (Paging is going through the roof).

My plan was to only start 100 of the simulators, measure memory usage, then start 100 more measure memory again and repeat until paging starts going up (In reality I will be taking more than three data points.) This should give me a figure for the amount of extra memory required for 100 simulators and enable me to project how much memory is required. I only need a rough idea +/-30Gb to avoid buying the full 2Tb ($150,000 worth) that the server will take. My question is whether this is a reasonable method to use and if so which Performance Counters would you monitor to give the amount of memory actually being used?

I am specifically talking about memory here as the difference between Working Set, Private Bytes, Committed, Shared, Virtual and all the other memory terms confuse me. I think I can manage to monitor CPU, IO and Networking by myself. The other thing I have noticed is that the .Net Cache adjusts its memory usage depending upon what is available which makes spotting a trend hard to see.

  • I'd be really wary of projecting memory usage based on 2 data points so close together. I would have serious doubts that memory usage (and I/O, for that matter) will scale linearly. It might, but I suspect it will tend toward non-linear as you move to large numbers. I'd test with a number of data points, progressing from small to large until performance becomes problematic (paging, I/O saturation, etc) and project from there. If it's possible to incrementally upgrade the machine and continue simulating w/ larger client numbers I'd do so until I got a good feeling for the shape of the curve. – Evan Anderson Feb 13 '12 at 14:58
  • You also need to give bit a better idea of what this is. Is it web? aspx? php? Something homegrown? Batch jobs? The behaviour of asp.net is different from loads of exes running on a box. You need some basic idea of what the system uses per user - approx. figures - and then an old envelope. How you get those figures depends on how your system works. – Ian Murphy Feb 13 '12 at 15:21
  • @Evan. I was always going to take more than two data points. – Martin Brown Feb 14 '12 at 8:48
  • @Ian: "some basic idea of what the system uses per user" is eactly what I am trying to find out. If I knew this I wouldn't need to ask the question. I have updated the question to try to cover your other points. – Martin Brown Feb 14 '12 at 8:50

Honestly? I DON'T.
When specing a server that will see any kind of real workload I cram in as much RAM as I can reasonably afford (systems are more likely to wind up RAM-constrained than CPU or Disk constrained - the only other guaranteed bottleneck is the front-side bus).

If you want to figure out how much RAM your application may use a basic load test like you've proposed is a good start, but if you already have this system in production (it sounds like you do) and your production system is swapping your task is easier: Figure out how much swap space you are using --> Add at least 2x that much RAM (round up to fit in your system's DIMM-size constraints).

If you perform a load test to get rough numbers and extrapolate from there remember to factor in a few things:

  1. The memory curve will probably be two distinct segments
    (initial sharp ramp up as frameworks/shared libraries are cached, then a slightly less steep curve as each new app's un-shareable code is put in memory)

  2. You still need free RAM for disk and shared library caching, and for the OS.
    (This should be at least a few gigs over what your app needs)

  3. ALL software leaks memory (at least all practical software does), so watch for that in your testing and be sure you have the room to deal with a leak.

  4. Your load will probably increase over the lifetime of the server. Plan accordingly.
    (If you don't have good capacity planning numbers, double today's workload and plan to handle that).

  5. Buying too much RAM today is cheaper than having your environment fall over tomorrow.

    • First Corollary: If you buy a slightly bigger server than you need you're the prescient admin who kept the company running. You will be largely ignored and unappreciated.
    • Second Corollary: If you under-size the machine and there are problems you're the incompetent buffoon who couldn't anticipate 500% growth, and everyone hates you.
  • I approve of the corollaries. – mfinni Feb 14 '12 at 18:49
  • 3
    @mfinni It's funny 'cuz it's true! – voretaq7 Feb 14 '12 at 19:27
  • Thanks for that. Though we have this system live, it is currently supporting a very small trial which means I can't get good figures off it. – Martin Brown Feb 15 '12 at 10:44

Thanks, the update at least gives everyone a clue. That you are contemplating 2Tb of memory means you're playing in a different ballpark to the usual setups. Big system. hate to think of how much heat thats going to be putting out.

Given that its an internal server process and that you are running out of memory (you don't say at what level you start paging) but I would want to eliminate the possiblity that the server process is consuming ever larger amounts of memory before going any further. If this is occuring it makes no difference what you do, the system will stop at some point.

I don't know of any generic tools which you can use to give you much more than a basic overview of whats going on... what comes with windows. The service process itself is a black box and your dev team need to provide monitoring tools.

Quick back of the envelope calculation:

2Tb of memory = 1024Gb = 1024*1024Mb = 1048576Mb
1048576Mb / 13000 connections = around 80mb per session

This would not be out of the range of a normal .net exe's working set.

Does the service have multiple threads? If they are launching a thread for each connection it would be worth looking at how they are doing this. ProcExp.exe from microsoft is an easy way to see if you have multiple threads and what those threads are consuming. It doesn't know about .net but will give you win32 counters.

Can you indicate how much memory and how many connections you had when doing your testing before it started paging?

So, how to establish if the server process has memory leak problems? It could be accumulating more memory with each session is connected, or it could be accumulating memory and not freeing them up ever.

What you could do is - pick a number of sessions which does not provoke paging and simulate that number of connections. - Run the simulation over a few hours and to use perfmon to watch the basic memory counters. - Repeat these tests with sessions which connect briefly and disconnect.

The idea being to see if the service is consuming more and more memory with each session, or if open sessions provoke ever increasing memory usage.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.