Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We are facing a weird problem in our data center. Our Backup server (running EMC Networker) loses network connection every alternate day around 3:00 AM (Backup schedule starts at midnight). After 2 hours of outage, the network connectivity recovers automatically and back to normal.

What we observed:

It is unlikely to be network issue, since it is directly connected to server farm switch (layer 2 connection without any intermediate hops). Further, the server is connected to two different switches for Load balancing using Broadcomm Teaming.

a) If it were a switch related issue it is unlikely that both the network ports go down, since they are connected to different switch.

b) A possibility Vlan wide issue is also ruled out since other devices in the same Vlan are fine.

c) Switch interface status is always up. But there are lot of packet drops during the outage period - Can be attributed to high interface utilization of the backup server (near 100%)

d) Connectivity is restored without any change on network.

Next suspect is resource utilization on Windows server. Both CPU and Memory have rarely exceeded 80%, but NIC card utilization is alarmingly high (near 100%)

Not really sure how to investigate this?

share|improve this question

5 Answers 5

I suspect driver problems or a duplex mismatch. Try to upgrade the drivers and check the duplex on each ends to be the same. Check also the ethernet statistics from the switch (if you have errors, collisions etc...).

What means loses network connectivity? It is not reachable, but the interface is up? It is reachable but has a lot of packet loss?

share|improve this answer
    
I checked on the duplex configuration and port speed settings. They match. The error rate and collision rate is minimal, but as I said earlier I see a lot of outDiscards. When I mean loss of connectivity, I mean all connections to device is lost. Packets targetted for the server drains into nothingness like a black hole.. does not reach the server, neither is the server able to send any packet across to the next hop. The server logs are barely helpful, only shows error messages associated with DC not reachable, NTP not reachable etc., –  Benny Jun 1 '10 at 14:50
    
Do you have monitoring on the server? Do you know the system/kernel time spend by CPU and the total CPU usage during the problem? If the kernel time is over 30% you should reinstall the NIC drivers. If you have output errors it could be that the frames were discarded because the output queue was overfilled. Check also the network equipment statistics. –  Mircea Vutcovici Jun 1 '10 at 21:27
    
Hmm.. I really didn't get a chance to dig out the root cause. The issue stopped happening and now everyone seems to have forgotten it. Classic infrastructure operations scenario. –  Benny May 7 '11 at 16:01

Did you try to change the network card?

share|improve this answer
    
Thanks for the responses guys. Update is that the server went back to normal operation without doing anything. Anyway we should have captured some perf logs as Mircea suggested lest the issue re-occurs. But the issue no longer happens and till then no point in capturing perf logs. –  Benny Jul 18 '10 at 13:40

How are you figuring that it loses network connectivity? What do you exactly mean by that? Do you have something monitoring it that tells you it can't be reached? Is there anything in the event logs, if so what?

If network utilazation is high, meaning to the point where it's fully saturated, you might see what appears to be dropped connection. However its tough to tell based on your description. My guess though is your backup job is maxing your pipe. Do you have a backup job that run every other day :)

share|improve this answer
    
The server was not reachable from the clients, neither could the server connect to the network as per event logs (unable to connect to DC etc.,). If high utilization is the culprit, is it possible that all connections would drop? Not even an ICMP can succeed? Is there anything you observed from your experience? –  Benny Jan 24 '12 at 8:48

This is a longshot, but maybe another device on your network has the same IP address? That would definitely cause traffic problems.

share|improve this answer
    
Sorry that was not the case! –  Benny Jan 24 '12 at 8:49

What kind of switch is it?

Are other devices on the same switch experiencing any communication problems when this specific problem is occurring? Though rare and improbable, you may be exhausting finite resources on that switch and in a sense DOS'ing yourself.

Also, if you have mechanisms in place to prevent DOS'ing, they could essentially blackhole'ing your problematic server until use-patterns return to normal.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.