There is a transaction processing server (TP-Server) to which my application (Client) connects to. We are seeing a strange packet sequence where in the TP-Server is sending a PUSH packet with a null (0) byte data, but this packet's sequence number is incorrect; the sequence number is that of the last received byte.
Please look at the TCP dump data here
http://pastebin.com/5UBXWazy
172.250.10.10.13824 is the TP-Server.
172.11.105.5.49524 is the Client.
"TP-Server" is not in my control, not sure what OS it is running on. "Client" is my payment application running on Debian Squeeze.
You can see that a null byte data is sent by TP-Server in the below packet.
20:52:34.472819 IP (tos 0x0, ttl 59, id 51820, offset 0, flags [DF], proto TCP (6), length 53)
172.250.10.10.13824 > 172.11.105.5.49524: Flags [P.], cksum 0xe36f (correct), seq 1457045850:1457045851, ack 3097912286, win 17680, options [nop,nop,TS val 646012206 ecr 73190886], length 1
0x0000: 4500 0035 ca6c 4000 3b06 a941 acfa 0a0a E..5.l@.;..A....
0x0010: ac0b 6905 3600 c174 56d8 c15a b8a6 63de ..i.6..tV..Z..c.
0x0020: 8018 4510 e36f 0000 0101 080a 2681 5d2e ..E..o......&.].
0x0030: 045c cde6 00 .\...
The sequence number of the above null byte data is marked as 1457045850. But the same was received much earlier in time as shown below.
20:50:22.506267 IP (tos 0x0, ttl 59, id 51817, offset 0, flags [DF], proto TCP (6), length 607)
172.250.10.10.13824 > 172.11.105.5.49524: Flags [P.], cksum 0x6460 (correct), seq 1457045296:1457045851, ack 3097912253, win 17680, options [nop,nop,TS val 645999009 ecr 73187775], length 555
For the incorrect sequence number, Client responds back with a sack flag indicating the incorrect sequence number.
20:52:34.472864 IP (tos 0x0, ttl 64, id 1172, offset 0, flags [DF], proto TCP (6), length 64)
172.11.105.5.49524 > 172.250.10.10.13824: Flags [.], cksum 0x4501 (correct), seq 3097912286, ack 1457045851, win 2003, options [nop,nop,TS val 73220891 ecr 646012206,nop,nop,sack 1 {1457045850:1457045851}], length 0
The same null byte data is sent repeatedly by TP-Server for 7 more times with the same incorrect sequence number. And Client deligently responds back with sack.
20:58:29.024492 IP (tos 0x0, ttl 59, id 51827, offset 0, flags [DF], proto TCP (6), length 53)
172.250.10.10.13824 > 172.11.105.5.49524: Flags [P.], cksum 0x04bf (correct), seq 1457045850:1457045851, ack 3097912308, win 17680, options [nop,nop,TS val 646047661 ecr 73277952], length 1
0x0000: 4500 0035 ca73 4000 3b06 a93a acfa 0a0a E..5.s@.;..:....
0x0010: ac0b 6905 3600 c174 56d8 c15a b8a6 63f4 ..i.6..tV..Z..c.
0x0020: 8018 4510 04bf 0000 0101 080a 2681 e7ad ..E.........&...
0x0030: 045e 2200 00 .^"..
20:58:29.024553 IP (tos 0x0, ttl 64, id 1179, offset 0, flags [DF], proto TCP (6), length 64)
172.11.105.5.49524 > 172.250.10.10.13824: Flags [.], cksum 0x602c (correct), seq 3097912308, ack 1457045851, win 2003, options [nop,nop,TS val 73309529 ecr 646047661,nop,nop,sack 1 {1457045850:1457045851}], length 0
However, the 8th time the same happens, Client stops giving ack. TP-Server continues to send null byte data for another 5 more times, for none of which Client responds with ack. This behaviour is happening at the network stack level and my poor Client application does not get any socket errors and only at time "21:18:59", it received a getsocketopt err:110 (which I think is ETIMED_OUT). So, my client application, waited for some more time and tried to reconnect at 21:21:38 (you can see the SYN packets). However, the TP-Server was not responding after that. After the "Client" PC was rebooted after a few mins, the TP-Server accepted new connections. But, the same null byte packets start to reappear again. At times, it has so happened, that the Client had some periodic data packets to be sent to the TP-Server at about 10 sec intervals and this kept the connection alive, irrespective of the null bytes still happening and client responding with sacks.
My Questions:
What is the reason behind these null data bytes sent with incorrect sequence number?
Should I do anything about the debian network stack to handle this scenario to keep the connection alive?