Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running Linux 3.2.51 in a virtual machine (192.168.33.15). I'm sending Ethernet frames to it. I'm writing custom software trying to emulate a TCP peer, the other peer is Linux running in the virtual machine guest.

I've noticed that TCP packets larger than about 4 KB are ignored (i.e. dropped without an ACK) by the Linux guest. If I decrease the packet size by 50 bytes, I get an ACK. I'm not sending new payload data until the Linux guest fully ACKs the previous one. I've increased ifconfig eth0 mtu 51000, and ping -c 1 -s 50000 goes through (from guest to my emulator) and the Linux guest gets a reply of the same size. I've also increased sysctl -w net.ipv4.tcp_rmem='70000 87380 87380 and tried with sysctl -w net.ipv4.tcp_mtu_probing=1 (and also =0). There is no IPv3 packet fragmentation, all packets have the DF flag set. It works the other way round: the Linux guest can send TCP packets of 6900 bytes of payload and my emulator understands them.

This is very strange to me, because only TCP packets seem to be affected (large ICMP packets go through). Any idea what can be imposing this limit? Any idea how to do debug it in the Linux kernel?

See the tcpdump -n -vv output below. tcpdump was run on the Linux guest. The last line is interesting: 4060 bytes of TCP payload is sent to the guest, and it doesn't get any reply packet from the Linux guest for half a minute.

14:59:32.000057 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [S], cksum 0x8da0 (correct), seq 10000000, win 14600, length 0
14:59:32.000086 IP (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 44)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [S.], cksum 0xc37f (incorrect -> 0x5999), seq 1415680476, ack 10000001, win 19920, options [mss 9960], length 0
14:59:32.000218 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0xa752 (correct), ack 1, win 14600, length 0
14:59:32.000948 IP (tos 0x10, ttl 64, id 53777, offset 0, flags [DF], proto TCP (6), length 66)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], cksum 0xc395 (incorrect -> 0xfa01), seq 1:27, ack 1, win 19920, length 26
14:59:32.001575 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0xa738 (correct), ack 27, win 14600, length 0
14:59:32.001585 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 65)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], cksum 0x48d6 (correct), seq 1:26, ack 27, win 14600, length 25
14:59:32.001589 IP (tos 0x10, ttl 64, id 53778, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [.], cksum 0xc37b (incorrect -> 0x9257), ack 26, win 19920, length 0
14:59:32.001680 IP (tos 0x10, ttl 64, id 53779, offset 0, flags [DF], proto TCP (6), length 496)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], seq 27:483, ack 26, win 19920, length 456
14:59:32.001784 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0xa557 (correct), ack 483, win 14600, length 0
14:59:32.006367 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1136)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 26:1122, ack 483, win 14600, length 1096
14:59:32.044150 IP (tos 0x10, ttl 64, id 53780, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [.], cksum 0xc37b (incorrect -> 0x8c47), ack 1122, win 19920, length 0
14:59:32.045310 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 312)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 1122:1394, ack 483, win 14600, length 272
14:59:32.045322 IP (tos 0x10, ttl 64, id 53781, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [.], cksum 0xc37b (incorrect -> 0x8b37), ack 1394, win 19920, length 0
14:59:32.925726 IP (tos 0x10, ttl 64, id 53782, offset 0, flags [DF], proto TCP (6), length 1112)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [.], seq 483:1555, ack 1394, win 19920, length 1072
14:59:32.925750 IP (tos 0x10, ttl 64, id 53784, offset 0, flags [DF], proto TCP (6), length 312)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], seq 1555:1827, ack 1394, win 19920, length 272
14:59:32.927131 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x9bcf (correct), ack 1555, win 14600, length 0
14:59:32.927148 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x9abf (correct), ack 1827, win 14600, length 0
14:59:32.932248 IP (tos 0x10, ttl 64, id 53785, offset 0, flags [DF], proto TCP (6), length 56)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], cksum 0xc38b (incorrect -> 0xd247), seq 1827:1843, ack 1394, win 19920, length 16
14:59:32.932366 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x9aaf (correct), ack 1843, win 14600, length 0
14:59:32.964295 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 104)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 1394:1458, ack 1843, win 14600, length 64
14:59:32.964310 IP (tos 0x10, ttl 64, id 53786, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [.], cksum 0xc37b (incorrect -> 0x85a7), ack 1458, win 19920, length 0
14:59:32.964561 IP (tos 0x10, ttl 64, id 53787, offset 0, flags [DF], proto TCP (6), length 88)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], seq 1843:1891, ack 1458, win 19920, length 48
14:59:32.965185 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x9a3f (correct), ack 1891, win 14600, length 0
14:59:32.965196 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 104)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 1458:1522, ack 1891, win 14600, length 64
14:59:32.965233 IP (tos 0x10, ttl 64, id 53788, offset 0, flags [DF], proto TCP (6), length 88)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], seq 1891:1939, ack 1522, win 19920, length 48
14:59:32.965970 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x99cf (correct), ack 1939, win 14600, length 0
14:59:32.965979 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 568)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 1522:2050, ack 1939, win 14600, length 528
14:59:32.966112 IP (tos 0x10, ttl 64, id 53789, offset 0, flags [DF], proto TCP (6), length 520)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], seq 1939:2419, ack 2050, win 19920, length 480
14:59:32.970059 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x95df (correct), ack 2419, win 14600, length 0
14:59:32.970089 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 616)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 2050:2626, ack 2419, win 14600, length 576
14:59:32.981159 IP (tos 0x10, ttl 64, id 53790, offset 0, flags [DF], proto TCP (6), length 72)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], cksum 0xc39b (incorrect -> 0xa84f), seq 2419:2451, ack 2626, win 19920, length 32
14:59:32.982347 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x937f (correct), ack 2451, win 14600, length 0
14:59:32.982357 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 104)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 2626:2690, ack 2451, win 14600, length 64
14:59:32.982401 IP (tos 0x10, ttl 64, id 53791, offset 0, flags [DF], proto TCP (6), length 88)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], seq 2451:2499, ack 2690, win 19920, length 48
14:59:32.982570 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x930f (correct), ack 2499, win 14600, length 0
14:59:32.982702 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 104)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 2690:2754, ack 2499, win 14600, length 64
14:59:33.020066 IP (tos 0x10, ttl 64, id 53792, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [.], cksum 0xc37b (incorrect -> 0x7e07), ack 2754, win 19920, length 0
14:59:33.983503 IP (tos 0x10, ttl 64, id 53793, offset 0, flags [DF], proto TCP (6), length 72)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], cksum 0xc39b (incorrect -> 0x2aa7), seq 2499:2531, ack 2754, win 19920, length 32
14:59:33.983810 IP (tos 0x10, ttl 64, id 53794, offset 0, flags [DF], proto TCP (6), length 88)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], seq 2531:2579, ack 2754, win 19920, length 48
14:59:33.984100 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x92af (correct), ack 2531, win 14600, length 0
14:59:33.984139 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x927f (correct), ack 2579, win 14600, length 0
14:59:34.022914 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 104)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 2754:2818, ack 2579, win 14600, length 64
14:59:34.022939 IP (tos 0x10, ttl 64, id 53795, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [.], cksum 0xc37b (incorrect -> 0x7d77), ack 2818, win 19920, length 0
14:59:34.023554 IP (tos 0x10, ttl 64, id 53796, offset 0, flags [DF], proto TCP (6), length 88)
    192.168.33.15.22 > 192.168.33.1.36522: Flags [P.], seq 2579:2627, ack 2818, win 19920, length 48
14:59:34.027571 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [.], cksum 0x920f (correct), ack 2627, win 14600, length 0
14:59:34.027603 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 4100)
    192.168.33.1.36522 > 192.168.33.15.22: Flags [P.], seq 2818:6878, ack 2627, win 14600, length 4060
share|improve this question
    
What's your implementation's initial congestion window? –  David Schwartz Oct 20 '13 at 13:51
    
@DavidSchwartz: There is no congestion window, my emulator sends some payload, and waits for a full ACK indefinitely. I know that a correct TCP implementation would resend eventually. But in this question I'm asking why I'm not getting an ACK from the Linux guest. Because it's not required to ACK everything is true but it doesn't answer my question: I'm interested in why it isn't ACKing in this particular case. –  pts Oct 20 '13 at 14:00
    
Does the TCP dump filter just show TCP traffic? –  Matthew Ife Oct 20 '13 at 15:02
    
No, it shows all traffic on the guest eth0. I used the command tcpdump -n -vv without any filter. –  pts Oct 20 '13 at 21:55

1 Answer 1

Well, this is meta-answer:

You didn't tell us which virtualization technology you were using, did you?

You have increased MTU inside properly, but we don't know whether inter-host/guest-working is actually properly working. I'd suggest for the beginning having something like Jumbo-frames enabled and verified it's working using very common tools, only after you verified it flawlessly working, you can put into the game your own TCP/IP-endpoint.

share|improve this answer
    
I ran tcpdump -n -vv in the internal eth0. I have all packages the guest has sent or received (see them in the question). I was running User-mode Linux with eth0=slirp,,/my/own/slirp, replacing the slirp binary with my own emulator. My most important question: why isn't the guest Linux sending an ACK when the TCP payload is that large? Why does the guest Linux keep silent? –  pts Oct 20 '13 at 21:59
    
I think that verifying everything else except your own piece of code is working okay is the first thing to try when you have to deal with such an unclear issue. As I already pointed out, there's "Jumbo Frames" which are sometimes an headache to implement in real networks, so unless you're sure they're working in your own setup, you can't be sure what exactly's going on and who's to blame. –  poige Oct 21 '13 at 7:11
    
As the tcpdump output shows, the guest Linux receives the TCP packet with the payload of 4060 bytes, but never sends an ACK. If I reduce the the payload size to 4010 bytes, I get an ACK (shows up in the tcpdump output and the host also gets it). ICMP works back and forth with frames of 50000 bytes. Knowing all these, could you please be more specific what I should try first, and what settings related to jumbo frames I should change? Is there some information I failed to provide, or is there another debug tool I should run? –  pts Oct 21 '13 at 10:10
    
What tcpdump actually does show it receives headers but it doesn't say the whole packet is really coming through network interface. Moreover, tcpdump shows responses from 33.15 have incorrect checksum… –  poige Oct 22 '13 at 1:20
    
How can a packet get lost after tcpdump in the guest has printed it? (There is no packet filtering set up in the guest.) I can try to fix the checksums. I didn't care much about them so far, because if I decreased the payload size by 50 bytes, everything went through. –  pts Oct 22 '13 at 8:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.