Route traffic to Kubernetes cluster using iptables

Question

I have a kubernetes cluster (currently only 1 node) running on my root server (netcup) and try to route traffic to it. Im pretty bad in networking so hopefully someone can help me.

On the cluster metallb is installed and calico as CNI. Metallb is responsible for assignment and announcement of IP addresses to K8s-LoadBalancers which works fine internally. So i can reach a load balanced service from within the server. I only expose a single service with a fixed internal IP (reverse-proxy)

Now I want to expose it to the outside by routing traffic directly to the IP address from metallb using iptables

The following rules were applied in iptables:

iptables -A PREROUTING -t nat -p tcp -m tcp -j DNAT --to-destination <metallb-ip> -i eth0 --destination-port 80 -m comment --comment Redirect web traffic to cluster (80)
iptables -A PREROUTING -t nat -p tcp -m tcp -j DNAT --to-destination <metallb-ip> -i eth0 --destination-port 443 -m comment --comment Redirect web traffic to cluster (443)
iptables -A POSTROUTING -t nat -p tcp -m tcp -j SNAT --to-source <public-server-ip> -o eth0 --destination-port 80 -m comment --comment Redirect web traffic from cluster (80)
iptables -A POSTROUTING -t nat -p tcp -m tcp -j SNAT --to-source <public-server-ip> -o eth0 --destination-port 443 -m comment --comment Redirect web traffic from cluster (443)

(maybe the commands are not 100% correct because i extracted them from my ansible script. You can find the ansible part at the bottom)

As far as i understand this should send all incoming TCP traffic to the internal IP using DNAT and all outgoing traffic will have the public server address using SNAT. So for a connecting client it should look like he is talking directly to the public IP/Port?

Unfortunately, trying to access the service from the Internet results in the following:

$> curl <public-server-ip>
curl: (7) Failed to connect to <public-server-ip> port 80 after 647 ms: Network is down

Access from the server works:

$> curl <metallb-ip>
404 page not found # which the expected answer since its a blank Traefik reverse proxy

An nmap port scan currently throws the following:

$> nmap -Pn -p 80,443 <public-server-ip>
Starting Nmap 7.93 ( https://nmap.org ) at 2022-12-15 15:57 CET
Strange SO_ERROR from connection to <public-server-ip> (50 - 'Network is down') -- bailing scan
QUITTING!

But it is up and running since im connected via SSH.

There is currently no ufw enabled.

I appreciate any help since as I said I am not very good at networking. ;)

Ansible script:

- name: DNAT port 80 to cluster
  iptables:
    table: nat
    chain: PREROUTING
    in_interface: eth0
    protocol: tcp
    match: tcp
    jump: DNAT # REDIRECT
    to_destination: <metallb-ip>
    destination_port: 80
    comment: Redirect web traffic to cluster (80)
  become: yes

- name: DNAT port 443 to cluster
  iptables:
    table: nat
    chain: PREROUTING
    in_interface: eth0
    protocol: tcp
    match: tcp
    destination_port: 443
    jump: DNAT # REDIRECT
    to_destination: <metallb-ip>
    comment: Redirect web traffic to cluster (443)
  become: yes 

- name: SNAT port 80 from cluster
  iptables:
    table: nat
    chain: POSTROUTING
    out_interface: eth0
    protocol: tcp
    match: tcp
    destination_port: 80
    jump: SNAT # REDIRECT
    to_source: <public-server-ip>
    comment: Redirect web traffic from cluster (443)
  become: yes

- name: SNAT port 443 from cluster
  iptables:
    table: nat
    chain: POSTROUTING
    out_interface: eth0
    protocol: tcp
    match: tcp
    destination_port: 443
    jump: SNAT # REDIRECT
    to_source: <public-server-ip>
    comment: Redirect web traffic from cluster (443)
  become: yes

- name: Configure IP Masquerading
  iptables: 
    table: nat
    chain: POSTROUTING
    out_interface: eth0
    jump: MASQUERADE

UPDATE: With the help of this article i tried to understand the flow. A added logging statements to investigate where thing are going wrong. That's what I found so far:

To check where packets get lost I added the following rules:

iptables -t raw -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "[raw:PREROUTING]: "
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "[mangle:PREROUTING]: "
iptables -t nat -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "[nat:PREROUTING]: "
iptables -t mangle -A POSTROUTING -p tcp --sport 80 -j LOG --log-prefix "[mangle:PREROUTING]: "
iptables -t nat -A POSTROUTING -p tcp --sport 80 -j LOG --log-prefix "[nat:PREROUTING]: "

Under /var/log/kern.log just the raw and mangle log statements appearing. So it seems that the nat-step is not triggered. Checking the mangle rules with: sudo iptables -t mangle -L but couldn't identify any issue here.

IP-Forward is enabled:

$> sudo sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

So why the packets don't come to the nat-step?

$> sudo iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
cali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
LOG        tcp  --  anywhere             anywhere             tcp dpt:http LOG level warning prefix "[mangle:PREROUTING]: "

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
cali-POSTROUTING  all  --  anywhere             anywhere             /* cali:O3lYWMrLQYEMJtB5 */

Chain KUBE-IPTABLES-HINT (0 references)
target     prot opt source               destination

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination

Chain cali-from-host-endpoint (1 references)
target     prot opt source               destination

Chain cali-to-host-endpoint (1 references)
target     prot opt source               destination

Chain cali-PREROUTING (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             /* cali:6BJqBjBC7crtA-7- */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere             /* cali:KX7AGNd6rMcDUai6 */ mark match 0x10000/0x10000
cali-from-host-endpoint  all  --  anywhere             anywhere             /* cali:wNH7KsA3ILKJBsY9 */
ACCEPT     all  --  anywhere             anywhere             /* cali:Cg96MgVuoPm7UMRo */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000

Chain cali-POSTROUTING (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere             /* cali:NX-7roTexQ3fGRfU */ mark match 0x10000/0x10000
MARK       all  --  anywhere             anywhere             /* cali:nnqPh8lh2VOogSzX */ MARK and 0xfff0ffff
cali-to-host-endpoint  all  --  anywhere             anywhere             /* cali:nquN8Jw8Tz72pcBW */ ctstate DNAT
RETURN     all  --  anywhere             anywhere             /* cali:jWrgvDQ0xEZHmta3 */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000