1

Let me get your thoughts on terminal server performance problems. The server hosts average 25 users which, after running some numbers, on average use 600MB memory with their main applications running (web browser, adobe reader, IP phone client). All users are on the same LAN as server. We constantly experience slow response and short session lockups. Combined CPU usage is on average 10%. What appears strange to me is that the system shows 29GB physical memory with 25GB of it free. The page file usage is about 50% averaging 9GB used.

Some server specs

OS: Server 2003 32bit Enterprise with /PAE flag RAM: 32GB CPU: 2xQuad Core @ 2.27Ghz HD: RAID5 1.2GB

After doing basic troubleshooting using performance monitor it leads me to believe that the performance problems are caused by the 32bit OS limitation in addressing full 32GB of physical memory even though the /PAE flag is used.

Can anyone suggest something, troubleshooting steps that can lead to a more conclusive answer?

Thanks

2
  • 1
    You might also be interested in this question. serverfault.com/q/332622/23300
    – Nic
    Oct 17, 2012 at 15:24
  • Have you had a look at the network yet? That has been my biggest challenge with RDP session performance. Look for congestion (too much broadcasting or flooding) and/or a large volume of TCP retransmits and duplicate ACK's. If you see any of these it's a good indication that you've got network issues that are impacting performance. When the problem occurs shadow a user from the server itself. If you're not seeing the same performance issues then the problem is most likely network related.
    – joeqwerty
    Oct 17, 2012 at 21:21

3 Answers 3

1

There is a bit of a little-documented annoyance with PAE in windows. Despite the fact that the OS is now capable of allocating all the RAM in the system, some applications still won't use it.

With PAE enabled, each process is still limited to the limits of a 32-bit environment (4gb)... unless it is specifically built to make use of AWE (Address Windowing Extensions).

Despite all this "info" ... I don't think this is the problem you're running into. (do you have processes exceeding 4gb of RAM?) 9GB of paging to me is quite a lot excessive. Doubly-so when you take into account that this is a terminal server. That much disk-IO is bad for performance. As mfinni said, it is more likely that you're running into issues with your disk IO than memory issues. I've seen the exact same symptoms you describe, only to find out that indeed my disk-drives couldn't keep up with the work-load. There are a laundry list of reasons for excessive Disk IO and just as many solutions to the problem.

In my case, I discovered that the print-spooler service was allocating HUGE amounts of RAM (which was mostly dumped to the paging file). It turns out there is a long-standing problem in the print-spooler whenever printers are created/deleted.

(when printers created, memory is allocated for the driver. When they're deleted, the memory is not de-allocated. The result on a terminal-server with users logging in & out all-day-long is a print-spooler service with 2gb+ of allocated RAM... and most of it thrown to the page-file) Where possible, do not use RDP to share the printers.

There are many other hidden gotchyas when it comes to terminal-services and memory.

2
  • This is interesting. Lets say if it was the print spooler, would restarting the service show some kind of memory/page file usage drop? How did you troubleshoot this problem? There is no single process that would be exceeding 4GB.
    – MikeM
    Oct 17, 2012 at 16:21
  • For a quick & dirty check... just open up the task-manager and look for "spoolsv.exe" and look at how much memory it's using. I saw almost 3gb once... it was rather scary. (server was up for months) I killed the process and saw the memory usage drop to almost nothing... and the system went non-responsive while the garbage-collection deallocated the swap-space. (huge amount of disk-io that caused the system to go non-responsive) After a minute or two... I restarted the service... and everything was running as it should be.
    – TheCompWiz
    Oct 25, 2012 at 16:23
1

Insufficient memory for user sessions may be doing it. Explain what you did in Perfmon that leads you to that conclusion, please?

edit - OK, I wouldn't worry so much about the paging file. It's different from how VM works on old Unix systems; Windows will more-aggressively page out things to keep more physical memory free. How to bring Paging File usage metric to zero? If you are really worried about paging, look at the page IO read rate. That's the hard fault rate.

A too-commonly-overlooked issue with interactice terminal servers is disk IO - that causes serious user experience issues without immediately jumping out at you from performance data. Does your RAID card have a BBWC, and are you doing write caching? If not, you're almost definitely seeing issues - using PerfMon look at the Disk Queue Length on the RAID volume. Rule of thumb (IIRC) is that a number higher than the number of physical spindles in the array is bad.

7
  • Simply looking at the Memory available bytes and Process working set (Total). Available memory is reported at 22GB. Working set total is at about 9GB. As mentioned before the page file is at about 9GB. There is no reason for the system to page so much when there is still extensive amount of physical memory available. (Numbers take at peak usage time). I should also add that the control has no BBWC and write caching is NOT enabled.
    – MikeM
    Oct 17, 2012 at 16:19
  • OK - based on that info, you can't say that you have a paging problem. It sounds like you don't, but i gave you more info to go on. And to repeat myself, fire up perfmon again and look at your disk queue length. If you're not doing write caching and are serving up desktops/standard office apps to that many users, I'd be shocked if that's not at least one of your problems.
    – mfinni
    Oct 17, 2012 at 16:30
  • Average disk queue length is under 0.5 with an occasional 1 and higher. This is a few minute sample. At what value is the disk queue length considered a bottleneck.
    – MikeM
    Oct 17, 2012 at 17:16
  • Keep that perfmon running and see what the numbers are when users start reporting problems. Maybe keep a desktop session on there yourself so you can more-easily identify when the problem happens. Rule of thumb (IIRC) is that a number higher than the number of physical spindles in the array is bad.
    – mfinni
    Oct 17, 2012 at 17:27
  • How do I troubleshoot this high page file? The difference between installed physical memory and physical memory available is way too big for the page file usage to be so high.
    – MikeM
    Oct 18, 2012 at 15:47
0

Windows 2003 x86 is definitely performance-challenged with regards to kernel memory. By default, it has lower paged pool maximum and non-paged pool maximum than Windows XP. We used to max out these two values, but even with that it is fairly easy to exhaust kernel memory.

More information here: https://serverfault.com/a/389299/20701

29 GB is really wasted on an x86 terminal server.

FYI, I would not rule out the network even though they are on the same LAN segment.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .