0

I have a setup on microk8s with cilium and Metallb where I run a docker registry pod. The pod is assigned a service that gets an IP address from metallb and everything works fine if I run curl against the repository from a client in a different VLAN. When I try to curl from the node itself I can only access the registry on the internal IP.

For context, this cluster has 4 nodes living in a VLAN and there is a lot of supporting infrastructure where the different functionality lives in their own VLAN. All traffic between VLANs goes through a router/firewall.

The cluster nodes are multihomed, i.e. they get an IP address in every VLAN (native VLAN and several tagged VLANs). If I give the registry an external IP from the native VLAN of the node again it all works fine and I can reach the registry using the external IP.

What I find strange is that when I shoot the curl request to the external VLAN the router sees the destination as the internal IP of the pod. This is where I got completely lost.

Again the external IP can easily be used from clients running on machines in different VLANs just not from the nodes themselves, anybody can help with what I could be missing?

metallb is configured to use Layer2 advertising (ARP).

I can provide more detail if needed, though at this point I am not sure where to start. tcpdump on the node does not register any traffic, while on the router it shows the destination as the internal IP of the pod. The router then has no idea what to do with it.

Edit -> adding more context:

tcpdump from the router, when reaching the registry pod from a remote machine:

IP 10.253.0.8.61859 > 10.9.40.146.5000: Flags [.], ack 2048, win 1025, length 0

From the node:

IP 10.9.40.148.13204 > 10.1.3.82.5000: Flags [S], seq 490446341, win 64240, options [mss 1460,sackOK,TS val 898569173 ecr 0,nop,wscale 7], length 0

So somehow instead of trying to reach the 10.9.40.146.5000, it tries to route to 10.1.3.82.5000.

The first IP is the IP of the service and an expected result, the other IP is the internal IP of the Pod, given out from the Cilium Cidr block (10.1.0.0/16)

Edit -> it also works if I do the call to the registry from the node, specifically using the nodes IP in the default VLAN:

curl --interface 10.9.41.52 -k -X GET https://10.9.40.146:5000/v2/_catalog

However when specifying just running:

curl -k -X GET https://10.9.40.146:5000/v2/_catalog it uses the VLAN 12 IP 10.9.40.148 as a source the problem occurs where the router sees the destination as 10.1.3.82 (internal pod IP)

Edit -> tcpdump on the node:

tcpdump -i ens2.12 src 10.9.40.148 and dst 10.1.3.82

gets traffic:

IP 10.9.40.148.4167 > 10.1.3.82.5000: Flags [S], seq 2235907990, win 64240, options [mss 1460,sackOK,TS val 907774853 ecr 0,nop,wscale 7], length 0

while doing the request:

curl --interface 10.9.40.148 -k -X GET https://10.9.40.146:5000/v2/_catalog

Edit -> routing policy I have setup

vlans:
    ens2.12:
        link: ens2
        id: 12
        dhcp4: false
        addresses:
            - 10.9.40.148/29
        routes:
            - to: default
              via: 10.9.40.145
              table: 12
        routing-policy:
            - from: 10.9.40.148/29
              table: 12

Thank you.

5
  • Are you using kube-proxy or Cilium's replacement?
    – pchaigno
    Commented Dec 10, 2023 at 11:36
  • Hi @pchaigno using kube-proxy in iptables mode.
    – Tomas
    Commented Dec 11, 2023 at 7:55
  • I see. Then AFAIK, kube-proxy should be in charge of externalIP service handling here, not Cilium. For the request that works, was it from a node inside the cluster or outside? If outside, did you try reaching the service from a remote node inside the cluster?
    – pchaigno
    Commented Dec 11, 2023 at 8:24
  • I tried all of the nodes. I double checked the routing table and can see the record I would expect: 10.9.40.144/29 dev eth2.12 proto kernel scope link src 10.9.40.148 The registry is on 10.9.40.146 and the node ip (where I got this table from) is 10.9.40.148
    – Tomas
    Commented Dec 11, 2023 at 8:28
  • Funnily enough there is another machine on the default VLAN that is NOT a kubernetes node and from there I can reach the registry without any issue. It is not multihome, only lives on the default VLAN, so it hits the router and reaches the service IP
    – Tomas
    Commented Dec 11, 2023 at 8:33

1 Answer 1

0

In case you tried to reach the service IP from the docker registry Pod itself check: Edge case: A Pod fails to reach itself via the Service IP.

1
  • Thank you for the help, however I am trying to reach the registry pod from the node, not from the pod itself.
    – Tomas
    Commented Dec 11, 2023 at 7:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .