We have a IIS web farm sitting in a DMZ connecting to a MSSQL DB(Windows Cluster). After troubleshooting some performance problems we stumbled upon strange network behavior.
We wanted to test network latency in general between the two servers, but ping is blocked by the firewall so we tried using a version of tcping to test latency specifically to the db server on port 1433. TCPing reports ~20ms latency. The same TCPing from my workstation to the DB server is ~2ms. My first thought was that this is the firewall between the Web and DB server, so, to confirm my suspicion, I ran another TCPing from another application server sitting inside the network and it ALSO reported the 20ms latency. Then I started running the same TCPing from some other servers. Some report 2ms latency some report nearly 20ms.
Network ops is telling me that the variances are most likely due to the OS configuration as the servers I tested are in the same network segments(with the exception of the web servers).
On servers where TCPing reports 2ms latency we see a dramatic improvement in performance.
Is there any type of network configuration in windows that could be causing this behavior? Does anyone have any other suggestions(other monitoring tools, other possible causes, etc)?
Update Just tried to TCPing to the local ip, not 127.0.0.1 but the machines actual ip and I also see latency(something like 15-18ms). I went around to several of the servers and notice similar behavior. This does not seem normal, any ideas? Not all servers exhibit this behavior.