Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a troublesome issue.

I have a 10Gb storage network serving a VMWare vSphere 5.1 environment.

  • Storage: TrueNAS(NFS) with a Chelseio T320 10Gb Nic
  • Storage: Dell PowerVault MD3600i(iSCSI)
  • Network: XSM7224S 10gb Switch
  • VMWare Hardware: Three Supermicro servers with dual hex proc and 192Gb RAM each.
  • VMWare 10gb Nics: Chelsio T420-LL-CR

What happens: Once a week to once a month, the storage network devices stop transmitting traffic and it all goes down in a ball of flame. None of the devices can ping each other on the storage network when this happens. It's as if the switch shut down all of the ports or stopped working, but, rebooting the switch does nothing. The network status within the linux and vmware systems report a status of "Up" but, regardless, restarting the network interfaces does nothing as well. I have to reboot all of the servers involved, including the TrueNAS but excluding the Dell Powervault, and then it comes back online and network starts flowing.

What I've done: This has been going on for a while and in that time I've replaced all the Nics(were 320's, upgraded to 420's), replaced the switch(was dell PC 8100), replaced the cables and added the Dell PowerVault MD3600i. The TrueNAS was acting as the primary storage but that job now belongs to the PowerVault and the TrueNAS is acting as storage for system backups. No logs have given any hint to what is occurring. The switch doesn't have any collisions or packet transmit errors to speak of. But regardless of all of this, the problem still occurs.

What I've yet to do: I'm going to replace the chelsio T320 NIC on the TrueNAS with a spare I have, tonight. I'm also going to put my Dell PowerConnect 8100 10Gb switch back in and segregate the TrueNAS network onto that switch simply because the TrueNAS is now currently the only constant in this problem that hasn't changed.

Am I missing something?: I'm at wits end with this and wanted to throw this problem at the community and see if I'm overthinking this or if there are some ideas to help determine the problem. I'm losing quite a bit of sleep and hair over this. I've seen bad Nic's take a network down but that almost always can be easily witnessed by seeing collisions on the switches port statistics.

Thank you! Brad

share|improve this question
    
The chelsio t320 driver has an issue where it suspends the card instead of reinitializing when it hits a fatal error. The root cause is really a hardware failure. Can you see any errors in logs (fatal errors/parity errors) relating to the cxgb interface? –  MaQleod Oct 7 at 19:46

1 Answer 1

I had a similar issue on a 1G network that we tracked down to flow control flaw with broadcom chipsets. During high PPS a nic would send a PAUSE frame to the switch. In PC 62XX and other broadcom based switches the default action is to propagate the PAUSE from to all ports sending traffic to the port that originally received the PAUSE frame. This can propagate to a switch completely shutting down at worst and at minimum you will see traffic blocked.

we ended up turning off flow control on all of our broadcom switches with 'no flowcontrol', since then we have not seen any network or partial network wide outages from high PPS.

share|improve this answer
    
+1, this was going to be my guess as well. –  Jed Daniels Mar 17 at 17:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.