How can I set up Kubernetes to forward a set of externally-facing ports on the master node to a service internal to the cluster?

Question

I have a Kubernetes cluster on which I'm running a ZeroMQ broker internally. I have set up a service so that this broker can be found by pods on the cluster, using the following Helm template:

#values.yaml
zmqServiceType:
  type: ClusterIP
  
zmqPub:
  port: 31339

zmqSub:
  port: 31342

zmqWsSub:
  port: 31343
----- 
#service.yaml
apiVersion: v1
kind: Service
metadata:
  name: zmq-broker
  labels:
{{ include "zmq-broker.labels" . | indent 4 }}
spec:
  type: {{ .Values.zmqServiceType.type }}
  ports:
    - name: zmq-sub
      port: {{ .Values.zmqSub.port }}
      protocol: TCP
    - name: zmq-pub
      port: {{ .Values.zmqPub.port }}
      protocol: TCP
    - name: zmq-ws-sub 
      port: {{ .Values.zmqWsSub.port }}
      protocol: TCP      
  externalIPs:
    #Master node IP external address 
    - 10.2.1.100 
  selector:
    app.kubernetes.io/name: {{ include "zmq-broker.name" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}

This works great within the cluster, pods are able to find the broker with no problem with the zmq endpoint "tcp://zmq-broker:".

However, I would also like to expose the zmq-broker so that it can be reached from outside the cluster.

I know I can do this very simply, by running kubectl port-forward:

kubectl port-forward <ZMQ_BROKER_POD> 31342:31342 31339:31339 31343:31343

But this has two disadvantages: It requires the kubectl port-forward command to be running all the time, and it requires one to know what pod the broker is running on a priori (plus to my understanding it is not meant for production).

What I would prefer is a way to map the outward facing ports not to the pod, but to the service, so that they will always find it no matter what the pod name is. It technically doesn't matter what physical node we connect to the internal service, but our preference is to make it the master node. We have no need for load balancing since there is only a single broker running in the system.

I had assumed all I needed was the externalIPs section in the services file, but that does not appear to be the case. The externalIPs section does indeed create a listening process on the master node, but it doesn't appear to forward TCP traffic.

Is what I'm attempting to do here possible, and if so, how?

Edit: I moved the external IP to one of the worker nodes, and it works fine, but no matter what IP I try on the master node, it won't forward traffic. Is there something special about the K8s master node that makes it not forward external IP traffic?

Answer 1

This turned out to be due to a routing problem.

The master node was unable to properly route to the worker nodes because flannel was using the wrong interface.

The Helm chart as written is correct. Once we fixed the routing issue everything worked.

If you find yourself in a similar situation where the master node doesn't work but the worker nodes do, the following can confirm that you have the same problem:

run

ip route 

On the worker nodes and the master node with some pods running. You should see pod IP addresses (10.244.X.X if using default CIDR) on both the head node and the worker nodes. In our case, we did not see the routes on the master node.

It was further confirmed to be a problem with flannel by looking at the logs for the flannel pods:

kubectl logs <Flannel_Pod_Name> --namespace kube-system

This showed errors when run for the head node pod, namely a "no route to host" error when it tried to create routing entries for the pods. The worker node logs showed flannel was successfully creating the routes.

The fix is to ensure that the default route on your master node passes through the device that has the IP address which communicates with your worker nodes (for us our default route on the master node originally was on a 192.X network, but should've been on a 10.X network for talking to the worker nodes).