We are facing a weird problem in our data center. Our Backup server (running EMC Networker) loses network connection every alternate day around 3:00 AM (Backup schedule starts at midnight). After 2 hours of outage, the network connectivity recovers automatically and back to normal.
What we observed:
It is unlikely to be network issue, since it is directly connected to server farm switch (layer 2 connection without any intermediate hops). Further, the server is connected to two different switches for Load balancing using Broadcomm Teaming.
a) If it were a switch related issue it is unlikely that both the network ports go down, since they are connected to different switch.
b) A possibility Vlan wide issue is also ruled out since other devices in the same Vlan are fine.
c) Switch interface status is always up. But there are lot of packet drops during the outage period - Can be attributed to high interface utilization of the backup server (near 100%)
d) Connectivity is restored without any change on network.
Next suspect is resource utilization on Windows server. Both CPU and Memory have rarely exceeded 80%, but NIC card utilization is alarmingly high (near 100%)
Not really sure how to investigate this?