Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

A couple of days ago we stumbled upon a disturbing issue with a relatively newly installed ESXi 5 management host for VDI. We were preparing a base VM for Linked-Clone deployment and when accessing its admin share from another machine ("\vm\c$") the entire Management network locks up. We can browse for a bit but after digging through a few folders Explorer hangs. The host and all other VMs inside of it are completely unreachable from the vSphere Client. If I physically walk over to the ESXi server I can login and reboot it and it will come back just fine. I can reliably crash it with any Windows-based VM (7 and 2008R2) 99% of the time. Today, I experimented with different physical ports on the server (there are 4) and found that once it crashes on a port, moving it to another and restarting the Management Network gets me back in, but if I fire up a share remotely I can crash that port, too. A reboot clears it all up.

I've combed through the logs on the server and haven't turned up anything of use. Any ideas?

share|improve this question
    
what type of server is the host? –  TheCleaner Aug 10 '12 at 17:49
    
What has changed the day or two before this started happening? –  jftuga Aug 10 '12 at 17:49
    
@TheCleaner It's a Dell R720. The VMs we're working with are all on local storage. –  silmaril8n Aug 10 '12 at 18:07
    
@jftuga Unfortunately, nothing changed. We setup the server over a month ago and loaded up a few VMs to get started. It's been sitting in the rack mostly idle though. –  silmaril8n Aug 10 '12 at 18:08
1  
Check the network switch ports for errors, collisions, retransmits, etc. –  jftuga Aug 10 '12 at 18:26

1 Answer 1

up vote 4 down vote accepted

After about an hour with VMware support we got down to the bottom of the issue. There is a known bug with Broadcom's Ethernet driver and VMware. By disabling NetQ the problem has, so far, gone away. I still see a few second delay when browsing into certain folders over the network, but it eventually loads and doesn't crash the NIC.

~ # esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:01:00.00 tg3         Up   1000Mbps  Full   24:b6:fd:f6:xxxx 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1  0000:01:00.01 tg3         Down 0Mbps     Half   24:b6:fd:f6:xxxx 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2  0000:02:00.00 tg3         Down 0Mbps     Half   24:b6:fd:f6:xxxx 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3  0000:02:00.01 tg3         Down 0Mbps     Half   24:b6:fd:f6:xxxx 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4  0000:42:00.00 ixgbe       Down 0Mbps     Half   90:e2:ba:0f:xxxx 1500   Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection
vmnic5  0000:42:00.01 ixgbe       Down 0Mbps     Half   90:e2:ba:0f:xxxx 1500   Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection

Count up the Broadcom/tg3 NICs (4 in our case).

~ # esxcfg-module -s force_netq=0,0,0,0 tg3

Reboot the host and you're done.

share|improve this answer
2  
It's always a Broadcom issue... –  ewwhite Aug 17 '12 at 15:47
    
@ewwhite We were slightly kicking ourselves for not going with the Intel card... there's a good chance we'll be swapping it out entirely now. –  silmaril8n Aug 17 '12 at 15:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.