2

A couple of days ago we stumbled upon a disturbing issue with a relatively newly installed ESXi 5 management host for VDI. We were preparing a base VM for Linked-Clone deployment and when accessing its admin share from another machine ("\vm\c$") the entire Management network locks up. We can browse for a bit but after digging through a few folders Explorer hangs. The host and all other VMs inside of it are completely unreachable from the vSphere Client. If I physically walk over to the ESXi server I can login and reboot it and it will come back just fine. I can reliably crash it with any Windows-based VM (7 and 2008R2) 99% of the time. Today, I experimented with different physical ports on the server (there are 4) and found that once it crashes on a port, moving it to another and restarting the Management Network gets me back in, but if I fire up a share remotely I can crash that port, too. A reboot clears it all up.

I've combed through the logs on the server and haven't turned up anything of use. Any ideas?

Improve this question
10
  • what type of server is the host?
    – TheCleaner
    Aug 10, 2012 at 17:49
  • What has changed the day or two before this started happening?
    – jftuga
    Aug 10, 2012 at 17:49
  • @TheCleaner It's a Dell R720. The VMs we're working with are all on local storage.
    – ceskib
    Aug 10, 2012 at 18:07
  • @jftuga Unfortunately, nothing changed. We setup the server over a month ago and loaded up a few VMs to get started. It's been sitting in the rack mostly idle though.
    – ceskib
    Aug 10, 2012 at 18:08
  • 1
    Check the network switch ports for errors, collisions, retransmits, etc.
    – jftuga
    Aug 10, 2012 at 18:26

1 Answer 1

Reset to default
4

After about an hour with VMware support we got down to the bottom of the issue. There is a known bug with Broadcom's Ethernet driver and VMware. By disabling NetQ the problem has, so far, gone away. I still see a few second delay when browsing into certain folders over the network, but it eventually loads and doesn't crash the NIC.

~ # esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:01:00.00 tg3         Up   1000Mbps  Full   24:b6:fd:f6:xxxx 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1  0000:01:00.01 tg3         Down 0Mbps     Half   24:b6:fd:f6:xxxx 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2  0000:02:00.00 tg3         Down 0Mbps     Half   24:b6:fd:f6:xxxx 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3  0000:02:00.01 tg3         Down 0Mbps     Half   24:b6:fd:f6:xxxx 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4  0000:42:00.00 ixgbe       Down 0Mbps     Half   90:e2:ba:0f:xxxx 1500   Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection
vmnic5  0000:42:00.01 ixgbe       Down 0Mbps     Half   90:e2:ba:0f:xxxx 1500   Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection

Count up the Broadcom/tg3 NICs (4 in our case).

~ # esxcfg-module -s force_netq=0,0,0,0 tg3

Reboot the host and you're done.

Improve this answer
2
  • 2
    It's always a Broadcom issue...
    – ewwhite
    Aug 17, 2012 at 15:47
  • @ewwhite We were slightly kicking ourselves for not going with the Intel card... there's a good chance we'll be swapping it out entirely now.
    – ceskib
    Aug 17, 2012 at 15:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .