4

There is a cluster on proxmox. In a cluster have vm - configured router (pfsense). In the our net have Windows and Linux machines.

There is the problem: Windows-machines have access to the internet, but linux-machines do not have.

All Linux machines behave the same way, it sends a request and waits for a response, but the response does not come back. I checked with wget, ssh and nslookup to external servers (8.8.8.8 example) and ping. All does not work, except ping. ICMP packets correctly go, hence I venture to suggest that the problem with TCP. But with all windows-machines, everything correctly works.

I was not create special filter rules for linux or windows.

I broke the head, but have not been able to solve this problem.

Baseline:

Our External IP:    XX.XXX.XX.XX
External Server IP:   Y.YYY.Y.YYY   (which was used to test)

And so, I spent two tests:

Test 1. wget from linux machine:

captured packets from router:

No.  Time         Source             Destination     Protocol Length Info
      1 0.000000  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     174 GET / HTTP/1.1
      2 21.798340 XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       74 49656 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=173036151 TSecr=0 WS=128
      3 21.850793 Y.YYY.Y.YYY        XX.XXX.XX.XX       TCP       74 http > 49656 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 SACK_PERM=1 TSval=1304117522 TSecr=173036151 WS=1024
      4 21.850849 XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       66 49656 > http [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=173036203 TSecr=1304117522
      5 21.850868 XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     174 GET / HTTP/1.1
      6 22.101939 XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     174 [TCP Retransmission] GET / HTTP/1.1
      7 22.605999 XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     174 [TCP Retransmission] GET / HTTP/1.1
      8 23.613997 XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     174 [TCP Retransmission] GET / HTTP/1.1
      9 25.629983 XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     174 [TCP Retransmission] GET / HTTP/1.1
     10 28.931894 Y.YYY.Y.YYY        XX.XXX.XX.XX       TCP       78 http > 11956 [FIN, ACK] Seq=1 Ack=1 Win=6 Len=0 TSval=1304124610 TSecr=172983238 SLE=109 SRE=110
     11 28.932001 XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       66 [TCP Previous segment not captured] 11956 > http [ACK] Seq=110 Ack=2 Win=115 Len=0 TSval=173043285 TSecr=1304124610
     12 29.661996 XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     174 [TCP Retransmission] GET / HTTP/1.1
     13 31.360021 XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     174 [TCP Retransmission] GET / HTTP/1.1

captured packets from my external server:

No.  Time         Source             Destination     Protocol Length Info
      1 0.000000  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       76 49656 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=173036151 TSecr=0 WS=128
      2 0.000059  Y.YYY.Y.YYY        XX.XXX.XX.XX       TCP       76 http > 49656 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 SACK_PERM=1 TSval=1304117522 TSecr=173036151 WS=1024
      3 0.059242  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       68 49656 > http [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=173036203 TSecr=1304117522
      4 0.059247  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     176 GET / HTTP/1.1
      5 0.303398  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     176 [TCP Retransmission] GET / HTTP/1.1
      6 0.809767  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     176 [TCP Retransmission] GET / HTTP/1.1
      7 1.815634  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     176 [TCP Retransmission] GET / HTTP/1.1
      8 3.836576  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     176 [TCP Retransmission] GET / HTTP/1.1
      9 7.088059  Y.YYY.Y.YYY        XX.XXX.XX.XX       TCP       80 http > 11956 [FIN, ACK] Seq=1 Ack=1 Win=6 Len=0 TSval=1304124610 TSecr=172983238 SLE=109 SRE=110
     10 7.134903  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       68 11956 > http [ACK] Seq=110 Ack=2 Win=115 Len=0 TSval=173043285 TSecr=1304124610
     11 7.864582  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     176 [TCP Retransmission] GET / HTTP/1.1
     12 9.564772  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     176 [TCP Retransmission] GET / HTTP/1.1

Test 2. wget from windows machine:

captured packets from router:

No.  Time         Source             Destination     Protocol Length Info
      1 0.000000  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       66 ampr-info > http [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
      2 0.053994  Y.YYY.Y.YYY        XX.XXX.XX.XX       TCP       66 http > ampr-info [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=1024
      3 0.054129  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       54 ampr-info > http [ACK] Seq=1 Ack=1 Win=65536 Len=0
      4 0.054450  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     150 GET / HTTP/1.0
      5 0.110034  Y.YYY.Y.YYY        XX.XXX.XX.XX       TCP       60 http > ampr-info [ACK] Seq=1 Ack=97 Win=6144 Len=0
      6 0.113011  Y.YYY.Y.YYY        XX.XXX.XX.XX       HTTP     931 HTTP/1.1 200 OK  (text/html)
      7 0.115990  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       54 ampr-info > http [RST, ACK] Seq=97 Ack=878 Win=0 Len=0

captured packets from my external server:

No.  Time         Source             Destination     Protocol Length Info
      1 0.000000  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       66 ampr-info > http [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
      2 0.053994  Y.YYY.Y.YYY        XX.XXX.XX.XX       TCP       66 http > ampr-info [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=1024
      3 0.054129  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       54 ampr-info > http [ACK] Seq=1 Ack=1 Win=65536 Len=0
      4 0.054450  XX.XXX.XX.XX       Y.YYY.Y.YYY        HTTP     150 GET / HTTP/1.0
      5 0.110034  Y.YYY.Y.YYY        XX.XXX.XX.XX       TCP       60 http > ampr-info [ACK] Seq=1 Ack=97 Win=6144 Len=0
      6 0.113011  Y.YYY.Y.YYY        XX.XXX.XX.XX       HTTP     931 HTTP/1.1 200 OK  (text/html)
      7 0.115990  XX.XXX.XX.XX       Y.YYY.Y.YYY        TCP       54 ampr-info > http [RST, ACK] Seq=97 Ack=878 Win=0 Len=0    

Please help me, any ideas?

7
  • To get a clue about what's going on, I'd do the following : Capture packets at the same time on Y.YYY.Y.YYY (use a home server, rent a vps, ... Anything external where you can have root access). If you can see incoming "GET / HTTP/1.1" your firewall likely drops related packets.
    – user130370
    Jan 11, 2014 at 20:17
  • Eric, thanks for the reply, Yes, Y.YYY.Y.YYY - this is really my vps. Second and fourth packet capture - I made from ​​it. I do see incoming "GET / HTTP/1.1" but also many other "[TCP Retransmission] GET / HTTP/1.1" I can not understand why my Y.YYY.Y.YYY not respond to this request, as in the case of windows: "http > ampr-info"
    – kvaps
    Jan 11, 2014 at 20:51
  • Oh thank you. Then the problem seems to be Y.YYY.Y.YYY. Y.YYY.Y.YYY : It should answer to "GET / HTTP/1.1". Something like "HTTP/1.1 200 OK". Anything in webserver logs?
    – user130370
    Jan 11, 2014 at 21:05
  • Hm, this task is more interesting than it sounds Unfortunately, not only Y.YYY.Y.YYY is no answer, but any server on the internet does not answer, even on the nslookup 8.8.8.8, same no answer. I used Y.YYY.Y.YYY for make the capture.
    – kvaps
    Jan 11, 2014 at 21:39
  • telnet Y.YYY.Y.YYY 80 from Linux fails also ?
    – krisFR
    Jan 12, 2014 at 0:46

4 Answers 4

1

I have seen a similar problem in the past. It turned out to be due to a problem with checksum offloading. The virtualization guest OS would offload checksuming to the virtualization host, which supposedly was supposed to offload it to the physical network interface.

But in the end the checksum was never computed, and the packet was sent over the network without a valid checksum. On the receiving end of the connection, the packet would be dropped due to having an invalid checksum.

On both the virtualization host and guest, you can try to look for options with any of the following names and try to turn them off:

  • UDP checksum offloading
  • TCP checksum offloading
  • UDP fragmentation offloading
  • TCP segmentation offloading
1
  • 1
    kasperd, you're right, helped enabling "Disable hardware checksum offload" option on System --> Advanced --> Networking --> Network Interfaces
    – kvaps
    Jul 3, 2014 at 14:15
1

Interesting issue. It doesn't look like a firewall problem, otherwise the initial 3-way TCP handshake wouldn't work. And despite the TCP retransmissions, it doesn't seem to be an MTU issue, as the packets are pretty small and, again, you're seeing them at both ends. It really looks like the web server is just not responding to the GET request.

The DNS lookups will be using UDP, so that's something else again. What do you see on a packet capture trying SSH? Also, what if you telnet to port 80 of your external server and issue a manual GET request:

telnet y.y.y.y 80
GET / HTTP/1.1
host: y.y.y.y
<line feed>

Try the above for HTTP 1.0 and 1.1. I notice that's a difference between the GET requests in the Linux and Windows packet caps. Shouldn't matter, but worth trying.

1
  • Unfortunately, telnet cannot even connect to port 80, ssh to port 22 is same
    – kvaps
    May 29, 2014 at 9:36
1

After a few months of work, I converted the network card driver on windows machine to virtio and the problem appeared also on the windows machine. That's true enough, it seems that the Problem is connected with personal incompatibility pfSense (or freebsd) and virtio-drivers or some other component of proxmox.

At this point, I decided to bypass this problem by:

  1. Proxmox cluster connected directly to the Internet and configure to router.
  2. Created another new subnet between proxmox and pfsense
  3. For each server, created a second network adapter in this new subnet and configured on each proxmox as the default gateway.
  4. On proxmox-cluster, created a rule that have to forward some ports to this servers and all external traffic to pfsense.

As a result I got the following:

  • All client computers on my network can access the Internet through pfSense, and use pfsense's services (proxy-server , logging , etc.)
  • All servers on proxmox cluster have access to the Internet directly through proxmox cluster
  • As a nice bonus, in the case if pfSense is down, I can still access the Proxmox cluster via SSH

Thank you all!

0

Unfortunately I'm unable to comment yet :-(
As far as I understand you have looked at the packets going through your router and to/from the target server. Have you tried tcpdump or wireshark on any of your misbehaving linux hosts to see what actually arrives/leaves the host?
Another thing to test might be to completely disable iptables on one of your linux hosts just for testing. Although you have not configured any special firewall rules on the hosts there might be a chance for some misbehaving default rules.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .