Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have distributed caching software (NCache, to be specific) that is deployed on two Windows 2003 x64 servers. The software is configured to run several caches, all in "replicated" mode (this is a typical active-passive failover clustering -- i.e. clients talk to one server, and that server copies all the data over to the second box so that if the first machine was ever to come offline, the second one would take over. Note, that this is not Windows clustering -- all the replication and failover is handled by NCache).

Everything worked fine for several months, but couple weeks ago we started seeing exceptions related to replication in the NCache log. The exceptions occurred once every several days, and did not appear to be predictable. NCache support basically said "check your network". Which we did. We found and resolved an issue (the details are too long to explain, the main point is that there was a high number of network collisions reported by both the network drivers on the second server, and the switch that server is connected to).

That was a week ago. After the network was fixed, the exceptions went away, and we thought that everything is OK. But today we got the exact same replication exception again. The only difference is that neither network drivers on the box nor the switch report any errors. I saw question http://serverfault.com/questions/76113/monitor-network-drops-between-windows-server-2003-servers, but don't think we can run wireshark on those boxes -- the network traffic on them is averaging about 100Mbit during the day, and since it's all NCache traffic, it's just not realistic to capture it all for days while waiting for the hiccup to occur. Is there anything else we can do to know for sure if it's still the network?

share|improve this question
add comment

1 Answer 1

Network troubleshooting can be difficult and finding the root cause of a network problem can take a lot of work. If you can't install and run Wireshark on the servers themselves you could install it on another machine and set up your switch (if it supports it) to mirror the server switch ports to the port that the Wireshark machine is connected to. A copy of every packet transiting the server switch ports will be copied to this port. You can configure Wireshark to save the capture data to multiple files at a set interval (time or size) and you can configure it to limit how much of each packet it captures. It's not likely that you need to capture each packet in it's entirety to see what's going on, you probably only need to capture enough of the packet to see what's going on at layers 1 through 3.

Don't neglect to look again at the physical layer. Clear the counters on the switch and then check the switch ports for collisions, errors, drops, etc. when a problem occurs. Look at the utilization of the relevant switch ports and look at the CPU and memory utilization of the switches.

Also look at perfmon counters for the NIC's in each server, specifically the Outbound Queue Length, Packets Outbound Errors, and Packets Received Errors counters.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.