Updated on 2014-10-09

Disclaimer

This question is similar to Unexplained slow gigabit network speeds but because the latter is still unanswered after 5 years, and that I think I'm able to add more details, I'm posting another question on this topic.

Background

We have 2 DELL PowerVault NX200 storage server. Both feature a Broadcom BCM5716 on-board dual Gigabit NIC (from which I'm using only one port). These storage server ship with Windows Server 2008 Storage Basic SP2, which I recently reinstalled and updated to the latest version.

We recently experienced slow network speed from the servers. I haven't tested the connection speed before so I can't tell what was the "normal" throughput. However, it is definitely very slow right now.

Issue

Throughput FROM both servers is around 5-6 Mbits/sec. This was measured with several runs of Iperf over the day. The throughput TO the servers is around 90 Mbits/sec.

Other tools and methods give the same kind of results (e.g. ttcp or copying files using Windows Explorer, Teracopy, ...). I will stick to Iperf results to exclude as many irrelevant components as possible (e.g. SMB).

What doesn't work

I tried several things, none of which worked.

  • Using different, brand new, network cables. I don't have the tools to properly test the quality of the cabling but I do get better speeds using the same cables on other computers.

  • Using a different NIC (BCM5709) which works fine on another Windows 2008 R2 server.

  • Unplugging all other computers and leaving the servers isolated on the switch.

  • Connecting the servers to a different switch didn't improve things.

  • Removing any switch and connecting directly NIC to NIC.

  • Updating the NIC drivers and software to the latest versions from QLogic.

  • Changing the NIC parameters to every "Speed & Duplex" mode. "Auto negotiate" would always result in the fastest speed I could get from it.

  • Same thing for others NIC parameters such as Jumbo Packet size (to match the switch support). Most of the time, it didn't change anything and when it did, it only made things worse.

  • I also tried all of netsh interface tcp set global parameters, including autotuninglevel which many people suggest to leave disabled.

  • Setting TCPWindowSize in the registry. This setting is ignored on Windows 2008.

What works

Booting to CentOS 6. Everything's fine on CentOS LiveCD. The culprit is probably Windows, the driver or the settings and not the NIC, switches or cabling.

(Probably) related issue

I was able to reproduce much of the results from Windows TCP Window Scaling Hitting plateau too early. Forcing the TCP window also results in much higher speeds during Iperf tests. It looks like Windows' TCP window scaling isn't working properly while transmitting.

Iperf results

Between the two servers, .12 NIC manually set to 100Mbps Full Duplex:

[  4] local 192.168.1.14 port 51535 connected with 192.168.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-15.1 sec   640 KBytes   348 Kbits/sec
[  4] local 192.168.1.14 port 5001 connected with 192.168.1.12 port 50498
[  4]  0.0-10.3 sec  6.00 MBytes  4.90 Mbits/sec

Between the two servers, .12 NIC manually set to 100Mbps Half Duplex (same results on "Auto"):

[  4] local 192.168.1.14 port 51541 connected with 192.168.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.2 sec  6.38 MBytes  5.23 Mbits/sec
[  4] local 192.168.1.14 port 5001 connected with 192.168.1.12 port 50509
[  4]  0.0-10.3 sec  6.12 MBytes  4.99 Mbits/sec
  • What's the NIC driver version, please? – Evan Anderson Oct 7 '14 at 23:44
  • Try booting from a Linux live CD to test the interfaces. Also, when you say 'FROM' the servers where is that connection going to? To another computer on the switch, or between each other? Also, have you tried directly connecting to the NIC (either between the two servers or from a laptop to one of the servers) without using a switch in between? – Gene Oct 7 '14 at 23:55
  • Also, have you tried using different NICs? If it has free expansion slots it's worth giving a try. Preferably an Intel NIC, but anything you have will do. If you're worried about the cost of buying NICs for each server just keep in mind the man hours you've spent on troubleshooting this will have cost a lot more than new NICs would. – Gene Oct 7 '14 at 23:56
  • More details: The current version of the NIC driver is 7.6.51. -- I get no more speed by connecting directly to the NIC. -- The computer-to-server network speed is normal, it's the server-to-computer speed that is the issue. -- I'm not concerned with the cost new NICs, but more for the times it takes to have them ordered and delivered. -- I was able to borrow a BCM5709-based NIC from another DELL server ...same result. -- I should have thought of trying Linux. Everything's fine on CentOS LiveCD. The culprit is probably Windows Storage Server 2008 and not the NIC. However, I'm out of ideas. – Pierre Oct 8 '14 at 1:02
  • In addition to checking the port/duplex settings on both sides of the connection, you should take a wireshark dump of an area you know is affected. Things like TCP errors due to congestion, and someone's illict DC++ server will be a lot more visible. Some things to look out for are: lots of black or red marked packets (duplicate or missing ACKS) - congestion, tons of ARP and IGMP, like way more than there should be - layer two loop, and other stuff I probably won't think of. Its a bit of a pain to figure out things like what percent of your traffic is arp, but math + counting ftw! – Some Linux Nerd Oct 8 '14 at 2:00

it might be an issue with the flow control on both the nic and switch, please see the following article and make sure you disable flow control on both the nic and switch.

http://www.smallnetbuilder.com/content/view/30212/54/

  • I second checking the physical switch flow control and speed/duplex settings. – SpacemanSpiff Oct 8 '14 at 1:30
  • I gave it a try quickly and it doesn't seem improve things. Still, I'll try to get a better understanding of this. – Pierre Oct 8 '14 at 1:32
  • It could def. be a half duplex on one side only issue. That makes stuff go reallllly slowly. – Some Linux Nerd Oct 8 '14 at 1:57
  • It seems like it. I get the same speed from Auto mode and 100Mbps Half Duplex. Yet, if I set the NIC to 100Mbps Full Duplex, iperf results are awful (no more than 500Kbits/sec). That is, only running Windows. On CentOS6 it's just fine. – Pierre Oct 9 '14 at 14:32

Sometimes strange network glitches can be caused by network filters (especially 3rd party firewall and/or antivirus software).

You can check which filters are installed (in powershell):

ls 'HKLM:\SYSTEM\CurrentControlSet\Control\Network\{4d36e974-e325-11ce-bfc1-08002be10318}' | 
    % { $_.GetValue('Description') } | 
    sort

or manually with regedit.

Also, check DPC (deferred procedure call) timing of network adapter drivers during iperf test. http://www.thesycon.de/deu/latency_check.shtml

Also, try running iperf (or preferably ntttcp) on loopback connection.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.