Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to figure out why tcp/ip throughput between two hosts I control is much lower than expected.

Host A: internet connectivity provided by Comcast (Residential) in Boston Metro -- can consistently achieve 100Mbps download from various well known sites (e.g. downloading Google Chrome)

Host B: internet connectivity provided by Cogent; can consistently achieve near 100Mbps to and from various sites.

Test Connection: Host A downloads a large file from B, via rsync/ssh on a non-standard port (22442) (i.e. rsync --progress -e 'ssh -p 22442' ... -- throughput is only around 2.5Mbps (292564 Bps)

Other clients (e.g a scratch EC2 instance) can achieve near 30 times throughput when attempting the same download from B

I've managed to capture statistics of the connection between A and B with tcptrace (attached below) but I'm not sure if anything seems wrong.

I imagine there is something on my end that needs tweaking, but I also have a slight suspicion about Comcast shaping traffic -- so before I start flailing around, I thought I would seek advice. What are some good next steps for me to try? Thanks in advance.

tcptrace -r -l -o1 dump.out
 1 arg remaining, starting with 'dump.out'
 Ostermann's tcptrace -- version 6.6.7 -- Thu Nov  4, 2004

 5763 packets seen, 5763 TCP packets traced
 elapsed wallclock time: 0:00:00.019828, 290649 pkts/sec analyzed
 trace file elapsed time: 0:00:18.655110
 TCP connection info:
 1 TCP connection traced:
 TCP connection 1:
    host a:        XXX.XXXX.XXX:41608
    host b:        XXX.XXXX.XXX:22442
    complete conn: RESET    (SYNs: 2)  (FINs: 1)
    first packet:  Sun May 26 14:48:52.923281 2013
    last packet:   Sun May 26 14:49:11.578391 2013
    elapsed time:  0:00:18.655110
    total packets: 5763
    filename:      dump.out
    a->b:                 b->a:
      total packets:          1946           total packets:          3817
      resets sent:              15           resets sent:               0
      ack pkts sent:          1930           ack pkts sent:          3817
      pure acks sent:         1874           pure acks sent:           35
      sack pkts sent:          228           sack pkts sent:            0
      dsack pkts sent:           0           dsack pkts sent:           0
      max sack blks/ack:         3           max sack blks/ack:         0
      unique bytes sent:      4100           unique bytes sent:   5457821
      actual data pkts:         55           actual data pkts:       3781
      actual data bytes:      4100           actual data bytes:   5457821
      rexmt data pkts:           0           rexmt data pkts:           0
      rexmt data bytes:          0           rexmt data bytes:          0
      zwnd probe pkts:           0           zwnd probe pkts:           0
      zwnd probe bytes:          0           zwnd probe bytes:          0
      outoforder pkts:           0           outoforder pkts:          23
      pushed data pkts:         55           pushed data pkts:         41
      SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/0
      req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y
      adv wind scale:            7           adv wind scale:            7
      req sack:                  Y           req sack:                  Y
      sacks sent:              228           sacks sent:                0
      urgent data pkts:          0 pkts      urgent data pkts:          0 pkts
      urgent data bytes:         0 bytes     urgent data bytes:         0 bytes
      mss requested:          1460 bytes     mss requested:          1460 bytes
      max segm size:           712 bytes     max segm size:          1448 bytes
      min segm size:            16 bytes     min segm size:            21 bytes
      avg segm size:            74 bytes     avg segm size:          1443 bytes
      max win adv:          301184 bytes     max win adv:           21632 bytes
      min win adv:            5888 bytes     min win adv:           14592 bytes
      zero win adv:              0 times     zero win adv:              0 times
      avg win adv:          269168 bytes     avg win adv:           21615 bytes
      initial window:           20 bytes     initial window:           21 bytes
      initial window:            1 pkts      initial window:            1 pkts
      ttl stream length:      4101 bytes     ttl stream length:        NA
      missed data:               1 bytes     missed data:              NA
      truncated data:            0 bytes     truncated data:            0 bytes
      truncated packets:         0 pkts      truncated packets:         0 pkts
      data xmit time:       18.128 secs      data xmit time:       18.527 secs
      idletime max:         5206.5 ms        idletime max:         5285.9 ms
      throughput:              220 Bps       throughput:           292564 Bps

      RTT samples:              57           RTT samples:            1661
      RTT min:                59.0 ms        RTT min:                 0.0 ms
      RTT max:               106.1 ms        RTT max:                40.1 ms
      RTT avg:                77.2 ms        RTT avg:                 1.4 ms
      RTT stdev:              17.2 ms        RTT stdev:               7.1 ms

      RTT from 3WHS:          60.0 ms        RTT from 3WHS:           0.0 ms

      RTT full_sz smpls:         2           RTT full_sz smpls:         1
      RTT full_sz min:        60.0 ms        RTT full_sz min:         0.0 ms
      RTT full_sz max:        70.1 ms        RTT full_sz max:         0.0 ms
      RTT full_sz avg:        65.1 ms        RTT full_sz avg:         0.0 ms
      RTT full_sz stdev:       0.0 ms        RTT full_sz stdev:       0.0 ms

      post-loss acks:            0           post-loss acks:           16
      segs cum acked:            0           segs cum acked:         2090
      duplicate acks:            0           duplicate acks:          201
      triple dupacks:            0           triple dupacks:            8
      max # retrans:             0           max # retrans:             0
      min retr time:           0.0 ms        min retr time:           0.0 ms
      max retr time:           0.0 ms        max retr time:           0.0 ms
      avg retr time:           0.0 ms        avg retr time:           0.0 ms
      sdv retr time:           0.0 ms        sdv retr time:           0.0 ms

A few updates I tried a few other things.

  • First I tried "rerouting" the flow by tunneling the transfer through a third host -- I achieved very respectable thoughput for about 30 minutes and then the throughput decayed by a factor of 20-ish to more or less match the original flow.

  • Second I ran mtr from B->A with packets of size 1400b, and saw the following:


    X.X.X       (0.0.0.0)                                                                        Mon May 27 19:20:37 2013
    Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                 Packets               Pings
     Host                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. X-X-X.X.X                                                               0.0%   173    0.1   0.1   0.1   0.2   0.0
     2. X-X.X.X-X.X.atlas.cogentco.com                                          0.0%   173    0.9   0.9   0.8   4.2   0.3
     3. X-X.X.X-X.X.atlas.cogentco.com                                          0.0%   173    0.7  12.7   0.7 211.8  38.3
     4. te7-3.ccr01.ord07.atlas.cogentco.com                                    0.0%   173    1.1  12.8   0.7 196.8  39.9
     5. te0-4-0-7.mpd21.ord01.atlas.cogentco.com                                0.0%   173    1.0   1.0   0.9   1.5   0.1
     6. be2006.ccr21.ord03.atlas.cogentco.com                                   0.0%   173    1.3   1.3   1.2   1.9   0.1
     7. be-202-pe04.350ecermak.il.ibone.comcast.net                             0.0%   172   39.7  41.6  36.3 156.7  15.4
     8. he-3-2-0-0-cr01.350ecermak.il.ibone.comcast.net                         2.3%   172   46.9  45.0  36.8  51.4   3.6
        he-3-1-0-0-cr01.350ecermak.il.ibone.comcast.net
        he-3-0-0-0-cr01.350ecermak.il.ibone.comcast.net
        he-3-12-0-0-cr01.350ecermak.il.ibone.comcast.net
     9. he-3-6-0-0-ar01.woburn.ma.boston.comcast.net                            1.2%   172   68.7  67.9  64.8  72.4   1.6
    10. X.X.X.X.boston.comcast.net                                              1.7%   172   67.6  66.8  64.2  70.4   1.0
    11. X.X.X.X.boston.comcast.net                                              1.2%   172   67.5  66.4  64.0  69.2   1.0
    12. X.X.X.X.comcast.net                                                     1.2%   172   75.7  75.2  72.0  85.8   1.9


Given the results of my first experiment, and the fact that so much packet loss is occurring between two routers controlled by Comcast in a Tier III facility (350 Cermak) -- I'm starting to lean more towards the "Comcast Boogeyman" explanation.

share|improve this question
    
Run a traceroute. you may be surprised how many routers there are in between your two servers. One or more of them will be the culprit. –  Matt May 27 '13 at 1:53
add comment

1 Answer 1

up vote 0 down vote accepted

I'd suggest a few things:

  • Check for packet loss with ping. Try at least 100 pings with different packet sizes, including a larger packet around 1400 bytes.

  • Round trip time (RTT) of ~70 ms could be quite high depending on distance. Maybe sure to look at a traceroute going each way. In the output you pasted, it appears that your traffic from a -> b is suffering from latency but not the other way around, which could suggest that there is a problem with the path from a -> b, but that the return path from b -> a is fine.

  • Look at the techniques in this old paper for analysing TCP throughput.

The paper is a little out of date (Ethereal is now Wireshark), but the techniques are still sound.

5 seconds of idle time compared to 18 seconds of wall clock time could suggest the connection going idle when a TCP window buffer (either send or receive) is getting full, which would contribute to poor throughput. However, the very asymmetric latency is the first thing that stands out, so I would focus on that first. Good luck and thanks for introducing me to tcptrace; I hadn't seen that before.

share|improve this answer
    
Awesome -- thanks for the pointers! I'll give all of those try and report back. –  lilinjn May 27 '13 at 0:48
    
The paper James suggested is great. The TL;DR: try to collect the pcap at B, and then look at the various TCP StreamGraphs in Wireshark –  lilinjn May 28 '13 at 0:13
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.