HyperV Cluster loses network connectivity randomly

Question

We've got the following setup

• 2 HP Gen9 64GB Ram hypervisors
• 1 PFSense Firewall
• 2 HP Switches
• QNAP NAS running VM Storage over 4 x 1GB bonded iscsi links.

The hypervisors have 5 VMs in the cluster basically running as a RDS setup. The setup is pretty new.

There are 4 1GB network cards in each hypervisor using nic teaming to make one 4GB connection for the VM usage / iscsi hosting of the VMS.

In the past week both servers at one time or another have dropped out of the cluster and gone offline, the cluster resumes on the other server but I would really like to know why it's going offline.

What happens is it just doesn't seem to be able to connect to the network in any way, the network settings are there but it cant' ping anything (internal or external), it can't tracert, can't see any other device on the network. I've tried disabling and re-enabling the nic team and the virtual Ethernet device but it doesn't come back online, the only way I can get the network restarted is to reboot the server.

But i can't for the life of me work out why it's happening

Answer 1

As @joeqwerty mentioned before, teaming the NICs for iSCSI network is not a good idea.

I'd recommend you to team two 1Gb NICs for the management network/live migration. Other two NICs, that will be used for iSCSI, should be separated with the Multipathing (MPIO) enabled.

When iSCSI networks are teamed, the Link Aggregation Control Protocol (LACP) processes each Ethernet frame, thus increasing latency and CPU load. So this could be the reason why the servers are getting disconnected.