I have a troublesome issue.
I have a 10Gb storage network serving a VMWare vSphere 5.1 environment.
- Storage: TrueNAS(NFS) with a Chelseio T320 10Gb Nic
- Storage: Dell PowerVault MD3600i(iSCSI)
- Network: XSM7224S 10gb Switch
- VMWare Hardware: Three Supermicro servers with dual hex proc and 192Gb RAM each.
- VMWare 10gb Nics: Chelsio T420-LL-CR
What happens: Once a week to once a month, the storage network devices stop transmitting traffic and it all goes down in a ball of flame. None of the devices can ping each other on the storage network when this happens. It's as if the switch shut down all of the ports or stopped working, but, rebooting the switch does nothing. The network status within the linux and vmware systems report a status of "Up" but, regardless, restarting the network interfaces does nothing as well. I have to reboot all of the servers involved, including the TrueNAS but excluding the Dell Powervault, and then it comes back online and network starts flowing.
What I've done: This has been going on for a while and in that time I've replaced all the Nics(were 320's, upgraded to 420's), replaced the switch(was dell PC 8100), replaced the cables and added the Dell PowerVault MD3600i. The TrueNAS was acting as the primary storage but that job now belongs to the PowerVault and the TrueNAS is acting as storage for system backups. No logs have given any hint to what is occurring. The switch doesn't have any collisions or packet transmit errors to speak of. But regardless of all of this, the problem still occurs.
What I've yet to do: I'm going to replace the chelsio T320 NIC on the TrueNAS with a spare I have, tonight. I'm also going to put my Dell PowerConnect 8100 10Gb switch back in and segregate the TrueNAS network onto that switch simply because the TrueNAS is now currently the only constant in this problem that hasn't changed.
Am I missing something?: I'm at wits end with this and wanted to throw this problem at the community and see if I'm overthinking this or if there are some ideas to help determine the problem. I'm losing quite a bit of sleep and hair over this. I've seen bad Nic's take a network down but that almost always can be easily witnessed by seeing collisions on the switches port statistics.
Thank you! Brad