1

There is a transaction processing server (TP-Server) to which my application (Client) connects to. We are seeing a strange packet sequence where in the TP-Server is sending a PUSH packet with a null (0) byte data, but this packet's sequence number is incorrect; the sequence number is that of the last received byte.

Please look at the TCP dump data here
http://pastebin.com/5UBXWazy

172.250.10.10.13824 is the TP-Server.

172.11.105.5.49524 is the Client.

"TP-Server" is not in my control, not sure what OS it is running on. "Client" is my payment application running on Debian Squeeze.

You can see that a null byte data is sent by TP-Server in the below packet.

20:52:34.472819 IP (tos 0x0, ttl 59, id 51820, offset 0, flags [DF], proto TCP (6), length 53)
    172.250.10.10.13824 > 172.11.105.5.49524: Flags [P.], cksum 0xe36f (correct), seq 1457045850:1457045851, ack 3097912286, win 17680, options [nop,nop,TS val 646012206 ecr 73190886], length 1
    0x0000:  4500 0035 ca6c 4000 3b06 a941 acfa 0a0a  E..5.l@.;..A....
    0x0010:  ac0b 6905 3600 c174 56d8 c15a b8a6 63de  ..i.6..tV..Z..c.
    0x0020:  8018 4510 e36f 0000 0101 080a 2681 5d2e  ..E..o......&.].
    0x0030:  045c cde6 00                             .\...

The sequence number of the above null byte data is marked as 1457045850. But the same was received much earlier in time as shown below.

20:50:22.506267 IP (tos 0x0, ttl 59, id 51817, offset 0, flags [DF], proto TCP (6), length 607)
    172.250.10.10.13824 > 172.11.105.5.49524: Flags [P.], cksum 0x6460 (correct), seq 1457045296:1457045851, ack 3097912253, win 17680, options [nop,nop,TS val 645999009 ecr 73187775], length 555

For the incorrect sequence number, Client responds back with a sack flag indicating the incorrect sequence number.

20:52:34.472864 IP (tos 0x0, ttl 64, id 1172, offset 0, flags [DF], proto TCP (6), length 64)
    172.11.105.5.49524 > 172.250.10.10.13824: Flags [.], cksum 0x4501 (correct), seq 3097912286, ack 1457045851, win 2003, options [nop,nop,TS val 73220891 ecr 646012206,nop,nop,sack 1 {1457045850:1457045851}], length 0

The same null byte data is sent repeatedly by TP-Server for 7 more times with the same incorrect sequence number. And Client deligently responds back with sack.

20:58:29.024492 IP (tos 0x0, ttl 59, id 51827, offset 0, flags [DF], proto TCP (6), length 53)
    172.250.10.10.13824 > 172.11.105.5.49524: Flags [P.], cksum 0x04bf (correct), seq 1457045850:1457045851, ack 3097912308, win 17680, options [nop,nop,TS val 646047661 ecr 73277952], length 1
    0x0000:  4500 0035 ca73 4000 3b06 a93a acfa 0a0a  E..5.s@.;..:....
    0x0010:  ac0b 6905 3600 c174 56d8 c15a b8a6 63f4  ..i.6..tV..Z..c.
    0x0020:  8018 4510 04bf 0000 0101 080a 2681 e7ad  ..E.........&...
    0x0030:  045e 2200 00                             .^"..

20:58:29.024553 IP (tos 0x0, ttl 64, id 1179, offset 0, flags [DF], proto TCP (6), length 64)
    172.11.105.5.49524 > 172.250.10.10.13824: Flags [.], cksum 0x602c (correct), seq 3097912308, ack 1457045851, win 2003, options [nop,nop,TS val 73309529 ecr 646047661,nop,nop,sack 1 {1457045850:1457045851}], length 0

However, the 8th time the same happens, Client stops giving ack. TP-Server continues to send null byte data for another 5 more times, for none of which Client responds with ack. This behaviour is happening at the network stack level and my poor Client application does not get any socket errors and only at time "21:18:59", it received a getsocketopt err:110 (which I think is ETIMED_OUT). So, my client application, waited for some more time and tried to reconnect at 21:21:38 (you can see the SYN packets). However, the TP-Server was not responding after that. After the "Client" PC was rebooted after a few mins, the TP-Server accepted new connections. But, the same null byte packets start to reappear again. At times, it has so happened, that the Client had some periodic data packets to be sent to the TP-Server at about 10 sec intervals and this kept the connection alive, irrespective of the null bytes still happening and client responding with sacks.

My Questions:

  1. What is the reason behind these null data bytes sent with incorrect sequence number?

  2. Should I do anything about the debian network stack to handle this scenario to keep the connection alive?

  • 1
    Whose bright idea was it to send such sensitive data across the public Internet unencrypted? If I understood your scenario correctly, you are seeing packets that appear to be retransmitting the last byte of the previous TCP segment, except that the actual value of that byte has changed in the meantime. And you expect that the intention was for them to send that byte with a sequence number one higher, such that it would come after the previous packet. I don't think there is much you can do without getting in touch with the administrator of the box at the other end of the connection. – kasperd Nov 14 '14 at 11:09
  • Once you have gotten in touch with somebody who can perform a tcpdump at the other end of the connection, the first thing to look for is whether the packets are modified in flight, or if they are originated corrupted in the first place. – kasperd Nov 14 '14 at 11:10
  • @kasperd "unencrypted" - yes, that's how the banks operate today world over. Banks have a protected private network and moreover, sensitive data like PINs are anyway encrypted. We are going off-topic. It is very difficult to ask the TP-Server folks to run tcp dump for us to debug the problem. I just wanted to ensure that Linux vs Windows OS have any incompatibility in handling such corrupted packets. – ReddyGB Nov 14 '14 at 11:24
  • I don't see how 172.11.105.5 could have responded any differently. It acknowledges the byte it received. The byte happens to have a different value than the first time it was received. But if that byte was already handed off to the application layer, the kernel wouldn't even remember what that byte was, so it couldn't even notice that the value is invalid. Also 172.250.10.10 has actually send ACK packets with the correct value of the next sequence number. So it is not like it has just been computed incorrectly at the sending end. – kasperd Nov 14 '14 at 11:56
  • Show them a dump of the traffic showing that you are receiving corrupted data from their system. If they want to refute that claim, they'll pretty much have to come up with a traffic dump showing otherwise. It might be a good idea to keep a tcpdump command running dumping all of the packets exchanged with them to a file, such that if they come back with a dump showing differently, you can compare the two captures to know exactly which packets are being corrupted in-flight. – kasperd Nov 14 '14 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.