Poor app performance on windows server 2008 R2

Question

I have a simple application that listens for TCP connections on a specific port and creates a new thread to handle each connection. The app is written in C#.NET 4.0 and uses the TCPListener object to handle the underlying socket connections.

I also have a simple load testing tool that creates a TCP connection to my service and then disconnects. I can set it to send thousands of requests all at the same time.

So, if I run the tcp server app on windows server 2003 I can send 10,000+ requests to it without any of them failing. But, when I run it on windows server 2008 R2 I can only send about 2,000 without any failures.

Here are the results of my tests for windows server 2008 R2...

D:\>EchoServerTest.exe -server server01 -port 8000 -connections 10000
Creating 10000 connections
All connections in progress
All connections complete in 20816ms
7252 established. 2748 failed.
Connection establishment errors
1693 - The semaphore timeout period has expired.
1055 - The remote computer refused the network connection.
Test Passed

Here are the results of my tests for windows server 2003...

D:\>EchoServerTest.exe -server server02 -port 8000 -connections 10000
Creating 10000 connections
All connections in progress
All connections complete in 3807ms
10000 established. 0 failed.
Connections reset
642 - An existing connection was forcibly closed by the remote host.
Test Passed

Two different OS's but same code being executed. Anyone have any ideas on why the performance is worse on 2008 R2?

Thanks.

Answer 1

Is the hardware of the two systems identical? When you test, are they using the same network port? It could be a bad port.

Could this be something to do with the SNP (Scalable Networking Pack) ? If you don't have the SNP applied on your 2k3 machine, but have it installed and with an outdated NIC driver on the 2k8 machine, you could have bad performance until you either update your driver, or disable TCP Chimney Offload .

Additionally, you could put some timing debug statements in your code to help determine exactly what operation(s) is or are taking longer on 2k8 - that would really help pin down what, in the OS, the problem is.

Answer 2

I figured out what was going on and how to fix it. The problem was TCP/IP port exhaustion. Here's the fix that solved my problem. (I found the following info from http://social.msdn.microsoft.com/Forums/en/winserver2008appcompatabilityandcertification/thread/36fd41b5-f57a-4401-af9b-328760ac5d38)

Increase the upper range of ephemeral ports that are dynamically allocated to client TCP/IP socket connections.

1) Start Registry Editor.

2) Browse to, and then click the following key in the registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

3) On the Edit menu, click New, DWORD Value, and then add the following registry value to increase the number of ephemeral ports that can by dynamically allocated to clients:

Value name: MaxUserPort

Value data: <Enter a decimal value between 5000 and 65534 here>

4) Close Registry Editor.

Note: You must restart your computer for this change to take effect.

Note: Increasing the range of ephemeral ports used for client TCP/IP connections consumes Windows kernel memory. Do not increase the upper limit for this setting to a value higher than is required to accommodate client application socket connections so as to minimize unnecessary consumption of Windows kernel memory.

I can now send 20000 requests to the 2008 R2 box and I get 0 failures. Thanks!