I've got two Linux VMs separated by a 50 mbps WAN connection. There is about 70ms of RTT on this WAN link, so it's got some latency. I've been testing my upload throughput to the Internet from each of these VMs. The basic setup is that Site A has an Internet connection and also a WAN connection to Site B. Site B uses the Internet connection at Site A.

I've noticed that Site B uploads a fair bit slower to the Internet than Site A. I was just using some of those Internet speed testing sites to test my upload speeds. I used the same Internet speed test site and server from each site to do the testing. I also ran the tests many, many times.

I ran some Wireshark caps to see what was going on. I assumed that the Internet server wasn't opening a wide enough TCP window to account for my 70ms of additional latency from Site B. However, the TCP window is wide open and it's actually my server at Site B that stops transmitting, waiting for ACKs to come in before sending more data.

I've looked at a bunch of stuff: TCP timestamps, SACKs, and Window Scaling are all enabled. I've increased my buffers as follows:

net.core.rmem_max = 67108864 
net.core.wmem_max = 67108864 
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.core.netdev_max_backlog = 30000

I've also increased by transmit queue length as follows:

ifconfig eth0 txqueuelen 10000

Lastly, I've disabled the software TCP segment offloading on the VM (there is no hardware TOE on my server).

Still though, Wireshark shows me that I don't get more than about 11,000 bytes in flight. I do have some dropped packets near the beginning of the transfer, but by the time things really get going, I would have expected more data to be flowing in flight.

Can anyone shed some light on why the sender is holding back data when so little data is actually in flight?

  • To utilize the full bandwidth with that latency, you would need to have 427KB in flight. Since a single TCP connection without window scaling can't have more than 64KB in flight, window scaling is pretty much a requirement for you. However if you only get 11KB in flight, then that's far less than what should be possible even without window scaling. But if window scaling options are corrupted in transit, then the maximum number of bytes in flight could end up far lower than 64KB, so that's the first problem I would look for. Do the SYN packets actually get through without any corruption? – kasperd Jan 19 '15 at 11:38
  • This is harder to answer, since one end of this is an Internet server that I don't have access to. However, my end shows me sending the scaling option, as well as receiving it from the server. I suppose it's possible that what I'm sending is getting stripped, but I'm not sure that would change anything: the server at the other end would not have scaling enabled since it didn't see it from me, but I would have scaling enabled since I have it set and saw it from the server. That means I should still try to fill the window unless there are dropped packets, and I don't see any of those. – Dave Robinson Jan 19 '15 at 19:14
  • Stripping the option on SYN packets from client to server would just mean scaling won't be used. Even if that happened, it wouldn't prevent you from having 64KB in flight. But if the window scaling option was corrupted, it could cause you to have much fewer bytes in flight. What is the scaling factor on the option you receive in the SYN-ACK packet? – kasperd Jan 19 '15 at 21:04
  • You are right that stripping the option on SYN "packets" would mean it's not is use. My host sends the option and it also receives the option, so my host is using scaling when calculating how large the receiver's window is. Whether my host's option got stripped on it's way to the receiver is something I can't see. That's what I meant. – Dave Robinson Jan 20 '15 at 14:33
  • To answer your question though, the scaling option that I receive is 8. I don't think this is the issue though, as the window is always wide open, with lots of room. The Wireshark cap also shows distinct pauses until ACKs come in before sending more data -- even when the window has tons of room. There is certainly lots of memory buffer space allocated too. Maybe I'm somehow hitting some sort of congestion control. I'm not sure how TCP figures this out. I'm using cubic right now for congestion control, anyone have a good resource on how this works? – Dave Robinson Jan 20 '15 at 14:36

What I see in your packet trace is congestion control reacting to packet loss.

The client starts out sending an initial 9 segments followed by slow start, where it send two more segments each time it receives an ACK packet.

The slow start algorithm continues until the first duplicate ACK from the server indicates that a packet has been lost. This happens at a point where there are 20820 bytes in flight. After this the client will grow the congestion window slower.

The second case of congestion happens only half a second into the transmission. After this the number of bytes in flight grows from around 15K and reaches 68012 bytes in transit by the time the third case of congestion happens which is 6 seconds into the transmission.

There is about 54KB in flight after the third case of congestion. This grows until it reaches 94384 bytes in flight, and the fourth case of congestion happens, this is 10 seconds into the transmission.

There are several more cases of congestion throughout the rest of the trace. The transmission might have been able to pick up speed, if it did not run into packet loss as often as it did. Having experienced the first packet loss as early as it did, it would take a long time reach full speed.

So what you need to figure out is why packets are lost that early during the TCP connection. It turns out that the packet lost at this point is one of the 9 packets send back-to-back at the very start of the connection. This indicates that the client is configured with a too high initcwnd setting. Judging from the one packet capture you provided, a reasonable setting would be 4.

The initcwnd setting is specified separately for each routing table entry. They can be viewed and changed using the ip route command.

  • And I think I know. When my interface is configured for 1000 mbps, the slow start ramps up quickly starts sending data very quickly, overwhelming the 50 mbps WAN link. However, when I set my interface to 100 mbps, slow start can't ramp up with nearly as much data, recuding my packet loss, and possibly preventing some buffer saturation in the WAN provider's router that has to transmit at 50 mbps. So here's what I've done, though I'm hoping for a better solution. I've used tc to rate limit traffic to the WAN connect subnets to 50 mbps. Now everything is fine when my interface is set to 1000mbps. – Dave Robinson Jan 23 '15 at 6:09
  • using tc is a bit of a hack to me though. Is there a way to get TCP not to overwhelm the 50 mbps circuit so quickly, while still performing decently on the local LAN? Or is there a way to tell TCP not to react so badly to duplicate ACKs? There are lots of duplicate ACKs because there is a 70ms RTT on the WAN link. Since it takes a while for my server to get the message, the remote end keeps sending dup ACKs. Any ideas? BTW: thanks for all your help. – Dave Robinson Jan 23 '15 at 6:15
  • I should stop writing this stuff so late at night too. I realized that my actual problem description is an amalgam of two problems that I'm having: one across a WAN link, and one to the Internet. I'm sorry for that confusion, that was completely my error. We've been discussing the WAN link issue. The WAN link is the 50 mbps link with 70 ms of latency. – Dave Robinson Jan 23 '15 at 6:21
  • One more addition. It seems that tc only helps for a bit (30 seconds when using CUBIC, 10 seconds when using reno). Seems that the tc's buffers are helping out for a bit, allowing me to generate a smooth 50 mbps of traffic. However, at some point, even the buffers get exhausted and I actually drop packets. When that happens, things slow down to about 15 mbps. I really need way to prevent TCP's congestion control from taking such drastic measures when it sees 20-30 duplicate ACKs. – Dave Robinson Jan 23 '15 at 7:09
  • @DaveRobinson I think your initcwnd setting is too high. See the updates to my answer. – kasperd Jan 23 '15 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.