1

On the host 1 physical adapter is plugged into our network, on host itself I have Hyper-V role. I have created a external switch and binded it to the physical Broadcom NetXtreme Gigabit Ethernet, box is checked for "Allow management operating system to share this network adapter".

I created a few VMs on this host, configured them to use my newly created switch and everything is working perfectly. At random, the virtual switch and physical switch stop receiving packets and all network activity stops functioning (I see packets being sent at times, nothing returned). At this point, I am unable to ping the server using IP nor am I able to communicate from host to outside world. The ugly workaround I have found is to disable/enable both physical NIC and virtual switch, at this point network connectivity resumes and I start receiving packets.

After doing some research I found that my NIC firmware was outdated, this is the version I found Network_Firmware_P32M4_WN64_7.6.15.EXE. The FW is installed using Dell SUU (SUU_730_64bit) and a reboot is also performed on server. At this point I feel I have found root cause but again as if my hardware is haunted, the problem happens again. Both physical/virtual adapters stop receiving packets and network connectivity is lost again.

> Driver details for Broadcom NetXtreme Gigabit Ethernet 
> Driver Date:01/28/2013 Driver Version: 15.6.0.10

Other things i have tried is manually setting duplex speed to full, still same issue. I have not messed with TCP offloading as I have been advised by many people that this is not related (something I could try if given valid reason?)

OS: Windows Server 2012 Datacenter

What could be the issue?

UPDATE Advanced properties of physical and virtual switch

NIC1                      802.3az EEE                    Disable                        *EEE            {0}
NIC1                      Flow Control                   Rx & Tx Enabled                *FlowControl    {3}
NIC1                      Interrupt Moderation           Enabled                        *InterruptMo... {1}
NIC1                      Jumbo Mtu                      1500                           *JumboPacket    {1500}
NIC1                      Large Send Offload V2 (IPv4)   Enabled                        *LsoV2IPv4      {1}
NIC1                      Large Send Offload V2 (IPv6)   Enabled                        *LsoV2IPv6      {1}
NIC1                      Maximum Number of RSS Queues   RSS 1 Queue                    *NumRssQueues   {1}
NIC1                      ARP Offload                    Enabled                        *PMARPOffload   {1}
NIC1                      NS Offload                     Enabled                        *PMNSOffload    {1}
NIC1                      Priority & VLAN                Priority & VLAN Enabled        *PriorityVLA... {3}
NIC1                      Receive Buffers                Default                        *ReceiveBuffers {200}
NIC1                      Receive Side Scaling           Enabled                        *RSS            {1}
NIC1                      Speed & Duplex                 Auto Negotiation               *SpeedDuplex    {0}
NIC1                      TCP/UDP Checksum Offload (I... Rx & Tx Enabled                *TCPUDPCheck... {3}
NIC1                      TCP/UDP Checksum Offload (I... Rx & Tx Enabled                *TCPUDPCheck... {3}
NIC1                      Transmit Buffers               500                            *TransmitBuf... {500}
NIC1                      Virtual Machine Queues         Enabled                        *VMQ            {1}
NIC1                      VMQ VLAN Filtering             Disable                        *VMQVlanFilt... {0}
NIC1                      Wake on Magic Packet           Enabled                        *WakeOnMagic... {1}
NIC1                      Wake on Pattern Match          Enabled                        *WakeOnPattern  {1}
NIC1                      EEE Control Policies           Optimal Power and Performance  EeeCtrlMode     {1}
NIC1                      Network Address                --                             NetworkAddress  {--}
NIC1                      VLAN ID                        0                              VlanID          {0}
NIC1                      Ethernet@WireSpeed             Enable                         WireSpeed       {1}
NIC1                      WOL Speed                      Lowest Speed Advertised        WolSpeed        {256}
vEthernet (extSwitch)     IPv4 Checksum Offload          Rx & Tx Enabled                *IPChecksumO... {3}
vEthernet (extSwitch)     IPSec Offload                  Auth Header and ESP Enabled    *IPsecOffloadV2 {3}
vEthernet (extSwitch)     Jumbo Packet                   Disabled                       *JumboPacket    {1514}
vEthernet (extSwitch)     Large Send Offload Version ... Enabled                        *LsoV2IPv4      {1}
vEthernet (extSwitch)     Large Send Offload Version ... Enabled                        *LsoV2IPv6      {1}
vEthernet (extSwitch)     TCP Checksum Offload (IPv4)    Rx & Tx Enabled                *TCPChecksum... {3}
vEthernet (extSwitch)     TCP Checksum Offload (IPv6)    Rx & Tx Enabled                *TCPChecksum... {3}
vEthernet (extSwitch)     UDP Checksum Offload (IPv4)    Rx Enabled                     *UDPChecksum... {2}
vEthernet (extSwitch)     UDP Checksum Offload (IPv6)    Rx Enabled                     *UDPChecksum... {2}
vEthernet (extSwitch)     Network Address                --                             NetworkAddress  {--}
  • Look for a setting on the Advanced tab of the adapter properties named "Energy Efficient Ethernet". If the setting exists disable it and see if that resolves the problem. – joeqwerty Jan 23 '14 at 0:05
  • No such value exists, see my edits in original post as I have included advanced properties for the NIC – KPS Jan 23 '14 at 0:17
  • I'm confused, do you have one or two physical nics in your box? Your question is unclear. If you have two, are they teamed? If so, which teaming mode? Have you tried breaking your team and run with just one nic to see if that helps? – Trondh Jan 23 '14 at 0:49
  • In reality, 4 physical NICS, all but 1 is enabled. This NIC is used for virtual switch. No NIC teaming. – KPS Jan 23 '14 at 1:03
  • I believe it's the EEE Control Policies setting (Energy Efficient Ethernet). Try disabling it or setting it to Maximum Performance. I had the same exact problem with the Intel NIC's in my Windows Server 2012 Hyper-V host and disabling EEE resolved it. – joeqwerty Jan 23 '14 at 6:47
2

Root cause is Virtual Machine Queue which can be found under the advanced properties of your network adapter. Disable it and all problems will go away. I have noticed this issue primarily with broadcom adapters but other adapters may be effected. Thorough explanation to come.

  • Thorough Explanation likely is "crap drivers". Sadly - i is something that repeatedly happens with Broadcom. Never had an issue with Intel. – TomTom Mar 30 '15 at 10:57
1

Just wanted to point out to official Microsoft KB about this: Virtual machines lose network connectivity when you use Broadcom NetXtreme 1-gigabit network adapters

As I understand this VMQ thing not properly implemented in cheap NICs and as vendors tend to deliver drivers which are targeted to whole product family of NICs not for specific one (i.e. one which includes slightly different chips and cards) they inclined to enable VMQ by default.
Solution is to disable VMQ as indicated in aforementioned KB, but it still lefts you with a problem caused by reverting this setting to ON after driver update. To address this problem you should implement some sort of configuration management in place. E.g. you can create configuration baseline in SCCM which includes VMQ=OFF criteria and monitor/automatically address configuration drift from your baseline caused by drivers update.
There is also series of very detailed blog posts on TechNet in case you want to know more about VMQ technology itself:
VMQ Deep Dive, 1 of 3

VMQ Deep Dive, 2 of 3

VMQ Deep Dive, 3 of 3

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.