3

I've been attempting to troubleshoot a network issue that presents with very high rate of TCP retransmits. 36 samples (taken with Wireshark 1.10.8 running on 32-bit Windows 7) totaling a little over seven hours, ranging between 2 and 53 minutes each shows retransmits occupying between 43 and 61 percent of the total ingress bandwidth.

What's confusing me is that as far as I know, there are only two reasons for this sort of issue: a flaky link that drops packets, and congestion. I believe that I have ruled these out. Let me lay out our situation, and I would love to hear from people more knowledgeable than myself on other directions of inquiry to resolve the problem.

The network in question is aboard a ship at sea. It uses a satellite link to communicate with the Internet. Unfortunately, the bandwidth costs for this type of link are prodigious, so we're stuck with a 1Mbps down / 512kbps up connection. Being a satellite link, it runs about 650ms ping times. At the moment, we have about 300 people aboard, all sharing that pipe.

The network consists of two VLANs (one for ship's computers, and the other for guests). Both VLANs are piped into a SonicWall TZ 215 (running SonicOS Enhanced 5.8.1.2-6o) which controls the pipe to the Internet. Both VLANs have wired and wireless clients. The wired network is run by a series of Cisco 2900 gigabit switches. The wireless network is provided by numerous Cisco APs (signal propagation in a steel ship at sea is terrible).

My first thought was that it was a congestion issue, so I pursued various solutions to this (blocking high bandwidth services like video chatting and streaming, bugging the corporate office to pay for a bigger pipe, etc). Sadly, we didn't get a bigger pipe. The other things helped a little bit, but not enough to make a real difference.

But this weekend I was put back to square one. The captain asked me to disable the guest access to the Internet during a drill. I took that opportunity to take a Wireshark capture of the network when it wasn't congested. To my surprise, that 10 minute sample showed the TCP retransmit rate was nearly identical to all the other captures - 58%. Over the duration of that sample, the average bandwidth usage was 98kbps, so it was definitely not congested.

This leaves just packet loss as a likely cause. To test this, I ran twelve hours of pings. At the end, the program reported less than 1% packet loss.

Which leaves... what? I don't know. Any additional ideas would be most appreciated.

  • What version of tcpip.sys do you have on the client (and target host if Windows)? – Greg Askew Aug 6 '14 at 13:08
  • The ping test was performed from a Windows 7 (32-bit) machine with tcpip.sys version 6.1.7601.22525 (11/26/13). The target of the ping was Google's DNS server, 8.8.8.8. I don't know what OS it runs, but I'm guessing it's not Windows – Erick Brown Aug 6 '14 at 13:15
  • Forgetting the Google ping test, what is the MTU of the link to the hosts where you are experiencing retransmits? Also, can you provide the output of the command: netsh interface tcp show global – Greg Askew Aug 6 '14 at 13:22
  • There are hundreds of clients on the network, all of them suffering from the same network issues. The mix includes Windows XP/7/8/Server 2k8, MacOS, iOS, and Android. That being said the typical MTU that I've seen from the Wireshark captures is 1514 – Erick Brown Aug 6 '14 at 13:27
  • A standard MTU on a LAN is 1500. If you subtract 28 from that, you should be able to ping a remote host with ping -f -l 1472 n.n.n.n with no timeouts or DF warnings. I have to say that "hundreds of clients" sharing a 1Mbit down/512 kbps up connection I would expect to see a lot of retransmits though. – Greg Askew Aug 6 '14 at 13:42
1

Check everything before your network. As in: The satellite link is flaky. Could be anything on the physical level of that side - bad calibration, whatever.

As per the Sherlock Holmes Approach that is the only thing left. Packets are lost because they are LOST.

  • If the satellite link was flaky, wouldn't 12 hours of pings shown more than 1% packet loss? Or am I thinking about that the wrong way? – Erick Brown Aug 6 '14 at 13:04
  • Yes. Ping packets are very small for example. Ping also may be faked. I just say - if you eliminate everything that is not it, then the remaining has to be it. And everything before your first ethernet cable all that is left. And that is the satellite link. – TomTom Aug 6 '14 at 13:11
  • That's a good point. Pings are small. An intermittent problem would be hundreds of times more likely to manifest in packets at the full MTU than they would be in a ping – Erick Brown Aug 6 '14 at 13:16
1

One good way to detect loss is by using a UDP stream of packets (there are various tools that do this, mainly for QoS testing). You can vary size, frequency, delay. It should show you if you have actual loss.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.