6

I have a Windows Server 2008 R2 x64 server that is AD controller and file server. I have a problem that Windows XP clients experience terribly slow (less than 10Mbps, really less than ten-megabits-per-second) downloading of files from a share.

Server is connected to 1Gbps switch using 1Gbps Nvidia NForce card and client is connected using 100Mbps built-in card.

This slow downloading can also be seen when I've booted client computer from CentOS Linux 5.5 Live-USB and used smbclient for downloading. But downloading from a Samba share on Linux server, which is also connected using 1Gbps link is fast.

What is very strange I've created a pair of programs (attached below) which test plain TCP throughput in C#, and they're performing as expected — at about 89Mbps.

I've disabled firewall on client and I'm using dot_nc_l 21000 > NIL on client and dot_nc [client_ip] < 100m.dat on Windows server. And I get about 9 seconds, when copying the same 100MB file from share takes over 2 minutes.

How to eliminate this?


Some pictures generated with wireshark on Linux client:

Downloading 100MB file from Windows 2008 CIFS file server connected with 1Gbps NIC to Centos 5 Linux client connected with 100Mbps NIC with smbclient: a graph looking like shark teeth

Downloading 100MB file from Fedora Linux CIFS file server on Samba connected with 1Gbps NIC to Centos 5 Linux client connected with 100Mbps NIC with smbclient (same scale as above): a graph looking like a seismograph not during earthquake


Here are these programs (linked are compiled using mono's gmcs, require .NET2):

dot_nc.cs

using System;
using System.IO;
using System.Diagnostics;
using System.Net.Sockets;

public class dot_nc
{
 public static void Main(string[] args) {
  string hostname = args[0];
  int port = int.Parse(args[1]);

  Stopwatch stopwatch = new Stopwatch();

  stopwatch.Start();
  TcpClient client = new TcpClient(hostname, port);
  stopwatch.Stop();
  Console.WriteLine("Connection: {0}ms", stopwatch.ElapsedMilliseconds);

  stopwatch.Reset();
  stopwatch.Start();
  byte[] buffer = new byte[4096];
  {
   Stream stdin = Console.OpenStandardInput();
   NetworkStream netout = client.GetStream();
   while ( true ) {
    int bytesread = stdin.Read(buffer, 0, buffer.Length);
    if ( bytesread <= 0 ) {
     break;
    }
    netout.Write(buffer, 0, bytesread);
   }
  }
  stopwatch.Stop();
  Console.WriteLine("Sending: {0}ms", stopwatch.ElapsedMilliseconds);
  client.Close();
 }
}

dot_nc_l.cs

using System;
using System.IO;
using System.Diagnostics;
using System.Net;
using System.Net.Sockets;

public class dot_nc
{
 public static void Main(string[] args) {
  int port = int.Parse(args[0]);

  TcpListener server = new TcpListener(IPAddress.Any, port);
  server.Start();
  TcpClient client = server.AcceptTcpClient();
  NetworkStream netin = client.GetStream();

  byte[] buffer = new byte[4096];

  Stream stdout = Console.OpenStandardOutput();
  int processed_bytes = 0;
  int processed_chunks = 0;
  while ( true ) {
   int bytesread = netin.Read(buffer, 0, buffer.Length);
   if ( bytesread <= 0 ) {
    break;
   }
   stdout.Write(buffer, 0, bytesread);
   processed_bytes += bytesread;
   processed_chunks++;
  }
  netin.Close();
  client.Close();
  server.Stop();
  Console.Error.WriteLine(
   "Received: {0} chunks of data of {1} average size", 
   processed_chunks, processed_bytes/processed_chunks
  );
 }
}
9
  • 3
    your tcp routine avoids the many layers of complexity that cifs adds
    – tony roth
    Dec 6, 2010 at 20:08
  • I know. But this complexity should not cause 10-fold slowdown, as this is just a little more work for CPU. Also this does not explain why uploads are fast and why downloads from Linux using CIFS are fast - complexity of these transfers are comparable.
    – Tometzky
    Dec 6, 2010 at 20:26
  • Did you try using a different NIC, switch, and cabling? It might be something as simple as hardware going bad.
    – Marcin
    Dec 7, 2010 at 16:08
  • What kind of switch is this? Dec 14, 2010 at 14:10
  • Cisco SR2024C: cisco.com/en/US/products/ps10016/index.html
    – Tometzky
    Dec 14, 2010 at 15:08

6 Answers 6

5

The problem was caused by:

  • too small packet buffers in inexpensive gigabit switch;
  • inadequate congestion avoidance algorithm used in Windows Server 2008 File Services;
  • disabled flow control in network adapter (it was disabled by default).

Because flow control was disabled, Windows was sending packets up to window size in one batch using 1Gbps connection. Since 100Mbps client receive packets much more slowly, almost all data up to window size needed to be buffered by a switch. As this cheap switch has very small buffers (buffer sizes aren't even stated in specifications but it has to be less than 64kB per port, as even disabling window scaling did not help) it had to drop excess packets. Packet loss caused a delay of about 0.25s seen on a graph. But congestion avoidance algorithm, used in File Services, or lack thereof, did not reduce TCP window size, so the next batch of packets wasn't smaller — it congested connection again and again causing congestion collapse.

Standard TCP connections (not File Services) must use different congestion control algorithm and do not get congested repeatably. I suppose treating File Services specially by Windows TCP stack helps in benchmarks against for example Samba.

So the solutions are:

  • Enable flow control in network adapter properties. It isn't an ideal solution, as any File Services transfer to 100Mbps client will also slow down concurrent transfers to 1Gbps clients to less than 100Mbps speeds.

  • Or connect 100Mbps clients to an enterprise class switch with much bigger buffers. This is a solution I've used. I have a 10 year old "3Com SuperStack 3 3300 SM" switch with one 1000Base-SX fiber optic gigabit Ethernet MT-RJ port. I bought a Cisco 1000BASE-SX mini-Gbic module (MGBSX1) with LC port for my Linksys gigabit switch and LC/MT-RJ multi-mode fiber patchcord (about $150 for both) and connected all 100Mbps clients to this 3com switch. I've also enabled flow control but it should not cause slowdowns with no 100Mbps client connected.

Thanks to SpacemanSpiff, whose comments helped to resolve this.

1

Does the Windows server have SMB Signing enabled? SMB signing adds slowness, and is enabled by default on Domain Controllers.

9
  • on top of that how about AV software?
    – tony roth
    Dec 6, 2010 at 20:07
  • I think it is enabled, as it's on by default. But I don't think it's a cause, as clients connected with 1Gbps link do not experience slow downloads from share. It would not slow down a server 10-fold. I'll test it when possible but I think that this is very unlikely the cause.
    – Tometzky
    Dec 6, 2010 at 20:16
  • No AV on server. Disabled on Windows XP client, no AV on Linux client. No high CPU utilization during transfer. The same client computer connected with 1Gbps is fast, so it's not AV.
    – Tometzky
    Dec 6, 2010 at 20:19
  • It certainly sounds like a higher-level issue. What about attempting other non-CIFS protocols - ftp, scp, tftp etc. from the slow client? Also don't discount multiple overlaid problems - Gig to 100full + SMB Signing/syncronous transfers + 8.3 filename + many small files with lots of metadata = death by a thousand cuts. Finally, perhaps look at the 2008 R2 Perf Tune guide, pp56 onward? microsoft.com/whdc/system/sysperf/Perf_tun_srv-R2.mspx
    – AndyN
    Dec 6, 2010 at 23:04
  • I've added graphs of transfers, good and bad, for comparison. Maybe something could be deduced from them.
    – Tometzky
    Dec 7, 2010 at 15:44
1

Might it be the 100Mbps card/switch? You mention that the same client works properly when it is on 1Gbps.

1
  • There are multiple clients. Some with 100Mbps cards, some with 1Gbps, some plugged in to 1Gb switch, some to 100Mb, which is plugged in to this 1Gb. All possible combinations. If there's any 100Mb link between a client and a server then CIFS performance is slower than 10Mb, like it is 1995 again.
    – Tometzky
    Dec 7, 2010 at 8:20
1

Feels like a lower-level network issue. My guesses:

  • Duplex mismatch issues. It certainly would bring performance down quite a bit. On the Linux side, use the ethtool command to verify that your negotiating at 100 Mbps/Full Duplex. If your card negotiates at 100/Half, and the switch thinks it the connection is 100/Full, then there will be all types of problems. You might want to experiment with forcing 100/Full instead of auto-negotiating the speed (remember you have to force 100/Full at the switch and the system)
  • It could also be a buffer issue either on the network card of the client, or the switch. I've seen Network card drivers not allocate enough buffer space, and cause issues with speed. I imagine the same type of problem could happen on the switch. Far harder to diagnose, other then swapping equipment.
6
  • The problem is when a server is connected with 1Gbps link, not 100Mbps. I have no other 1Gb switch to test. Also this does not explain why simple TCP transfer is fast. I really do not understand this as CIFS also simply uses TCP.
    – Tometzky
    Dec 13, 2010 at 9:17
  • Reading the problem description tells us that the issues aren't physical, as plain TCP transfers easily obtain full speed. Dec 14, 2010 at 0:08
  • 2
    I completely forgot to take into consideration that straight transfers over raw TCP operate normally. I noticed that your in your raw TCP Transfer, your are redirecting the output on the receiving end to the null device. Have you tried redirecting it to an actual file? Could we be looking at Disk IO propblems?
    – SteveM
    Dec 14, 2010 at 14:38
  • This is not IO problem, as transfers 1G->1G are fast. Also 100M->100M are fast. Only 1G->100M by CIFS are slow. I was suspecting switch buffers also - to confirm this I benchmarked raw TCP and got fast transfers. This is just mad.
    – Tometzky
    Dec 14, 2010 at 23:04
  • I was thinking more Disk IO problems. Your scripts really only test network performance, and disk I/O from the sending system. Just trying to rule out disk IO performance issues on the receiving end.
    – SteveM
    Dec 22, 2010 at 21:42
0

You could try to copy via esetutil (if you have an exchange server there)

Check this: http://blogs.technet.com/b/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

This could be an test if you copy large files from client to server or vice versa to test if esetutil get better performance.

also i had similar Problems with a Windows 2008 and Linux Server with an option named NetDMA (last section). This solved my problems (was an Broadcom Network Adapter with teaming)


How to enable and disable NetDMA in Windows Server 2008 To enable or disable NetDMA, follow these steps: Click Start, click Run, type regedit, and then click OK. Locate the following registry subkey, and then click it: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters Double-click the EnableTCPA registry entry. Note If this registry entry does not exist, right-click Parameters, point to New, click DWORD Value, type EnableTCPA, and then press ENTER. To enable NetDMA, type 1 in the Value data box, and then click OK. To disable NetDMA, type 0 in the Value data box, and then click OK. If the EnableTCPA registry entry does not exist, enable the NetDMA functionality. The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these product


from Support Microsoft KB 951037

0

Check if the BIOS of your servers/clients has a "CPU C State" parameter (probably inside the Power Saving category). If present, try setting that parameter to DISABLE.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .