1

We have a IIS web farm sitting in a DMZ connecting to a MSSQL DB(Windows Cluster). After troubleshooting some performance problems we stumbled upon strange network behavior.

We wanted to test network latency in general between the two servers, but ping is blocked by the firewall so we tried using a version of tcping to test latency specifically to the db server on port 1433. TCPing reports ~20ms latency. The same TCPing from my workstation to the DB server is ~2ms. My first thought was that this is the firewall between the Web and DB server, so, to confirm my suspicion, I ran another TCPing from another application server sitting inside the network and it ALSO reported the 20ms latency. Then I started running the same TCPing from some other servers. Some report 2ms latency some report nearly 20ms.

Network ops is telling me that the variances are most likely due to the OS configuration as the servers I tested are in the same network segments(with the exception of the web servers).

On servers where TCPing reports 2ms latency we see a dramatic improvement in performance.

Is there any type of network configuration in windows that could be causing this behavior? Does anyone have any other suggestions(other monitoring tools, other possible causes, etc)?

Update Just tried to TCPing to the local ip, not 127.0.0.1 but the machines actual ip and I also see latency(something like 15-18ms). I went around to several of the servers and notice similar behavior. This does not seem normal, any ideas? Not all servers exhibit this behavior.

0

4 Answers 4

2

Bull, network ops is incompetent.

2ms is not small but ok.

20ms is outrageous. That is a WAN link, or an overloaded line or something.

NO LAN technology in a building will give you 20ms, even if you go through half a dozen routers.

I am not aware of any malconfiguration.

Blocking ICMP though can cause side effects - like dropped TCP packets. Whoever turned that off should learn about TCP/IP before configuring firewalls. By TCP standards, the network is broken due to missing ICMP (which is used for example to find the maximum segment size that can safely be transported).

4
  • I wish it were a WAN link, at least it would be explainable to our users! :) Thanks, I will ask them to open up ICMP see what happens.
    – Dan
    Apr 24, 2013 at 12:24
  • ICMP has now been enabled but I don't see any improvements.
    – Dan
    Apr 25, 2013 at 6:04
  • Out of luck - ops must find the bottleneck. 20ms is not reasonable in a WAN scenario, and you have no way to figure out where the bottleneck is. They have to.
    – TomTom
    Apr 25, 2013 at 6:06
  • See my update, TCPing to local machine also latent...
    – Dan
    Apr 25, 2013 at 8:15
1

OS Configuration? I've seen that sort of latency when the switch port and the NIC were not agreeing on speed and duplex. That is set within the OS on the NIC side. However, if that's occurring, they should be getting errors on the switch ports in question.

Where OS configuration would come into play with respect to performance is with SMB, as that does differ between OS versions and OS configurations can cause, say, Windows Server 2003 and Windows Server 2008 to not play nicely together. However, that isn't related to network latency and that's a completely different protocol than what you use to connect to SQL Server or likely what is used by TCPing.

1
  • We checked speed and duplex and everything is set to auto negotiate. Networks does not see any issues on the switches so that is not it.
    – Dan
    Apr 25, 2013 at 6:05
1

It's in the hardware either the hardware itself is insufficent or misconfigured. it's layer one or two, or firewall ruleset good luck with them.

When you tcping SQL from an IIS server, capture the traffic using wireshark installed on SQL. Capture everuything and then thin it out with the display filter. or you can do a capture filter like: TCP port 1433 and right click a packet and do follow TCP stream...

Have your FW admins look at this: http://support.microsoft.com/kb/968872

There's more than just 1433 to open and it sounds like they're not very good at this in which case their ruleset is suspect, icmp ingress from DMZ should be blocked. It should not be used. inside a core where your workstation and SQL are is fine. I'm guessing your network is logically two layers, a public where your IIS sits and core where SQL is.

By segment, do they mean subnet or vlan? If it's VLAN then there are rules for there.... you can have the "segment" open to TCP 1433, and not UDP or you have a vlan a host is missing or not in there that should be, instead it's in another.

did you configure OS to not do computer browser service, WINS, stuff like that so it's not trying to ID your SQL by netbios? That could slow things down.

I would open a wireshark capture on your SQL box and see what you see. You can turn off autoscroll packets in live capture so they don't go screaming by. Let it capture do some transactions then stop it and use the display filter so drill down, do like tpc.port eq 1433 I think. the capture filter would be: port 1433 and that gets all protocols only destined to and from 1433 on the subnet on which that machine is a member of.

You'll most likely see a lot of Retransmits, duplicate Acks and things of that nature. Look at ARP, see if that's going ok, if you see netbios traffic do what you have to turn it off in os. Like on IIS nic you have windows client unchecked or file sharing in network stack, I forget exactly off top of head. You want only TCP and UDP btwn sql and IIS. no NetBT, NetBios or whatever. ALL DNS, no WINS, etc... good luck with your network staff.

0

I would try disabling tcp chimney offload http://www.iislogs.com/steveschofield/troubleshooting-iis-7-network-performance-issues-and-tcp-chimney-offload-receive-side-scaling-and-network-direct-memory-access

1
  • Tried this actually already and it didn't help
    – Dan
    May 6, 2013 at 9:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .