I've got two Linux VMs separated by a 50 mbps WAN connection. There is about 70ms of RTT on this WAN link, so it's got some latency. I've been testing my upload throughput to the Internet from each of these VMs. The basic setup is that Site A has an Internet connection and also a WAN connection to Site B. Site B uses the Internet connection at Site A.
I've noticed that Site B uploads a fair bit slower to the Internet than Site A. I was just using some of those Internet speed testing sites to test my upload speeds. I used the same Internet speed test site and server from each site to do the testing. I also ran the tests many, many times.
I ran some Wireshark caps to see what was going on. I assumed that the Internet server wasn't opening a wide enough TCP window to account for my 70ms of additional latency from Site B. However, the TCP window is wide open and it's actually my server at Site B that stops transmitting, waiting for ACKs to come in before sending more data.
I've looked at a bunch of stuff: TCP timestamps, SACKs, and Window Scaling are all enabled. I've increased my buffers as follows:
net.core.rmem_max = 67108864 net.core.wmem_max = 67108864 net.ipv4.tcp_rmem = 4096 87380 33554432 net.ipv4.tcp_wmem = 4096 65536 33554432 net.core.netdev_max_backlog = 30000 net.ipv4.tcp_congestion_control=cubic
I've also increased by transmit queue length as follows:
ifconfig eth0 txqueuelen 10000
Lastly, I've disabled the software TCP segment offloading on the VM (there is no hardware TOE on my server).
Still though, Wireshark shows me that I don't get more than about 11,000 bytes in flight. I do have some dropped packets near the beginning of the transfer, but by the time things really get going, I would have expected more data to be flowing in flight.
Can anyone shed some light on why the sender is holding back data when so little data is actually in flight?