1

On a large private network there seems to be excessive latency between two boxes which I control. This is stopping the server providing a web service to the client as its causing timeouts.

Here's a slow running request logged in Charles

Timing  
Request Start Time  03/11/10 23:21:33
Request End Time    03/11/10 23:21:33
Response Start Time 03/11/10 23:21:42
Response End Time   03/11/10 23:21:42
Duration    8.99 sec
Request Duration    16 ms
Response Duration   0 ms
Latency 8.97 sec
Speed   1.30 KB/s
Response Speed  ∞ KB/s
Size    
Request Header Size 412 bytes
Response Header Size    151 bytes
Request Size    -
Response Size   11.17 KB (11436 bytes)
Total Size  11.72 KB (11999 bytes)
Request Compression -
Response Compression    - 

As a test I just tried getting an 423k image file via a browser, but this downloads so slow it stalls.

How can I troubleshoot where the problem lies?

I tried using pingplotter alt text

Looks like hops 3 and 4 are the culprits? Where do I go from here?

ANSWER

This turned out to be a problem with the client box running Windows Web Server 2008 R2 and a known network speeds problem if the router is not able to handle network TCP window scaling. Updated router software and problem was resolved.

1
  • 3
    First, isolate the test to the server itself. Does the delay occur if you try to download the same image from the server itself? If so, it's a server problem. If not, it's a network problem.
    – joeqwerty
    Nov 4, 2010 at 0:07

4 Answers 4

2

Attempt to replicate the issue from as many different points as possible. You'll probably find something in common between them (be it the server they are using, a switch, etc). Collect traceroute data from each endpoint.

It's always better to use the onion approach: add/remove layers. You can start form the client and come closer and close to the server. Or the inverse.

Make sure to check the interface statistics for error counts. I find it very useful to fire up tcpdump/Wireshark on both ends and capture the packets from the TCP session, then compare both. Wireshark does a good job of pointing out the most obvious problems (like checksum errors or retransmissions).

5
  • 2
    Personally I would take the opposite approach. Start from the "nucleus" and work your way out. The server is the "nucleus", so try to replicate the problem on the server itself, if everything works as expected then move out one layer to a host connected to the same switch, then move out another layer, etc. If on the other hand, you try to replicate the problem from multiple external points you'll wind up chasing a lot of wild geese. The key is to isolate and troubleshoot.
    – joeqwerty
    Nov 4, 2010 at 2:17
  • Joe's right. Your approach gives you no common denominator to work with, so you will waste a LOT of time testing bits that may or may not have a bearing on the problem. Nov 4, 2010 at 3:15
  • bear in mind, that network conditions , such as load can change throughout the day, or be effected by external conditions. You need a network performance baseline to compare your results with. If you can't measure it when it's 'working', how do you measure it when it's not? Nov 6, 2010 at 13:53
  • I said "add/remove layers" meaning start from the center or the borders. Your choice. Networks differ a lot and sometimes people don't have enough access to use one or the other approach. But good point.
    – gtirloni
    Nov 6, 2010 at 22:04
  • Added traceroute data to question, just not sure where to go next?
    – Saul
    Nov 7, 2010 at 0:40
1

This turned out to be a problem with the client box running Windows Web Server 2008 R2 and a known network speeds problem if the router is not able to handle network TCP window scaling. Updated router software and problem was resolved.

http://www.intel.com/support/motherboards/server/sb/CS-030717.htm

0

Looking at your pingplotter results, it looks as if the main problem displayed is atht some devices in the path are slow to respond to ICMP echo requests (if #10 is your end device, you're getting avg 42 ms RTT, but you have devices in the middle that ave an avg 82 ms RTT, this is possibly down to load on the device, but it can be hard to tell).

Is the server requiring name resolution before serving content? There seems to be 9 seconds from "request made" to "request acknowledged" and the only thing that really match that would be if the server is trying to do a lookup (of some sort) and not immediately getting a response.

4
  • Some devices in the path didnt respond to ICMP requests, so I set it to TCP on port 443 (because its an https connection Im trying to sort out). If I run the same scan using the server iP instead of domain name I get the same results
    – Saul
    Nov 8, 2010 at 13:41
  • The interesting bit isn't if the client end has DNS issues, it's if the server end has it.
    – Vatine
    Nov 8, 2010 at 15:20
  • How would I test for that Vatine? As far as I know the server doesnt ask for any name resolution.
    – Saul
    Nov 8, 2010 at 15:38
  • Depends on how "from the ground up" you want to go. I'd, personally, start by doing a tcpdump of a request/response pair, see if there are any associated requests going out from the server and take it from there. Bit hard to say, though, without knowing the application, at all.
    – Vatine
    Nov 9, 2010 at 9:55
0

install mtr and run it between client and server and vice a vesa. You'll be able to see packet loss and latency per hop. Very useful tool. Beware that if you network is shaping traffic, then results will not be reliable.

alt text

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .