While trying to open a TCP Port to host B from A, I figure out there's a problem in the 3 way TCP handshake between the two hosts.
Client/Host A send a SYN to host B, Host B send a SYN/ACK, then host A send a RST packet.
I sniffed packets in wireshark, and found that the SYN/ACK packet does not have the same TCP checksum after it has been sent by A and received by B. Otherwise it is the exact same packet. Wireshark mark the packet as "ACKed unseen segment".
I do not have any hops between the two machines. The two hosts are VMs in VMware ESXi hosts . The VMs are in different hosts. They are directly connected in the same VLAN in ESXi. The client VMs is on ESXi v4.1.0. I opened all the firewall for my tests. And finally I totally turned off the TCP checksum offload. I did the same test with a VM that is on the same host, same result. The client VM runs debian Jessie 8.8, and the other VMs are debian wheezy.
I'm suspecting ESXi is modifying the checksum between the two machine.
As I don't have much experience with VMware ESXi, and debugging it, I'd like to know how to "disable" the checksum TCP? It would be a workaround for now and test that the problem really come from the checksum.
I tried to add a iptables mangle rule like that on host A:
iptables -t mangle -A PREROUTING -s #IP_HOST_B# -o eth0 -j CHECKSUM --checksum-fill
But doesn't change anything.