We are running two separate subdomains, each on a separate external IP address, and each matched to its own kubernetes nginx service. The configuration looks like this:
#-------------------- # config for administrative nginx ssl termination deployment and associated service apiVersion: extensions/v1beta1 kind: Deployment metadata: name: admin-nginx labels: name: admin-nginx spec: replicas: 1 template: metadata: name: admin-nginx labels: name: admin-nginx spec: nodeSelector: cloud.google.com/gke-nodepool: currentNodePool containers: - name: admin-nginx image: path/to/nginx-ssl-image:1 ports: - name: admin-http containerPort: 80 - name: admin-https containerPort: 443 apiVersion: v1 kind: Service metadata: name: admin-nginx spec: ports: - name: https port: 443 targetPort: admin-https protocol: TCP - name: http port: 80 targetPort: admin-http protocol: TCP selector: name: admin-nginx type: LoadBalancer #-------------------- # config for our api's nginx ssl termination deployment and associated service apiVersion: extensions/v1beta1 kind: Deployment metadata: name: public-nginx labels: name: public-nginx spec: replicas: 2 strategy: rollingUpdate: maxUnavailable: 0 template: metadata: labels: name: public-nginx spec: nodeSelector: cloud.google.com/gke-nodepool: currentNodePool containers: - name: public-nginx image: path/to/nginx-ssl-image:1 ports: - name: public-http containerPort: 80 - name: public-https containerPort: 443 apiVersion: v1 kind: Service metadata: name: public-nginx spec: ports: - name: https port: 443 targetPort: public-https protocol: TCP - name: http port: 80 targetPort: public-http protocol: TCP selector: name: public-nginx type: LoadBalancer #--------------------
Inside of our kubernetes cluster, we have, associated with each of the nginx deployments, a custom API router/gateway we use internally. These routers each have a /health endpoint, for, ah, health checks. This will be important in a second.
Some of the detail above have been elided; there is also a bit of configuration that makes nginx aware of the target service's address and port.
The configuration above creates 2 load balancers, sort of. I guess, technically, it creates two forwarding rules, each with an associated external IP, and a target pool consisting of all the instances in our k8s cluster. This, generally speaking, should work fine. Each k8s generated forwarding rule has an annotation in its description field, like this:
An associated firewall entry is created as well, with a similar annotation in its description field:
The external IPs are then wired up to one of our subdomains via CloudFlare's DNS service.
Ideally, the way this should all work, and the way it had worked in the past, is as follows.
An incoming request to admin.ourdomain.com/health returns the health status page for everything handled by the API router deployment (well, the service pointing to the pods that implement that deployment anyway) that deals with admin stuff. It does this by way of the nginx pod, pointed to by the nginx service, pointed to by the description annotation on the forwarding rule, pointed to by way of the GCE external IP address manager and the firewall, though I'm less clear on the ordering for that last part.
server status lookupMicros https://adminservice0:PORT/health Ok 910 https://adminservice1:PORT/health Ok 100 https://adminservice2:PORT/health Ok 200 https://adminservice3:PORT/health Ok 876
And so on.
Meanwhile, a request to public.ourdomain.com/health should return pretty much the same thing, except for public services.
server status lookupMicros https://service0:PORT/health Ok 910 https://service1:PORT/health Ok 100 https://service2:PORT/health Ok 200 https://service3:PORT/health Ok 876
Pretty reasonable, right?
As best I understand it, the whole thing hinges around making sure a request to the admin subdomain, by way of the external address linked to the admin annotated forwarding rule, eventually make its through GCE's network apparatus and into the kubernetes cluster, somewhere. It shouldn't matter where in the cluster it ends up first, as all of the nodes are aware of what services exist and where they are.
Except... That's not what I'm seeing now. Instead, what I'm seeing is this: every couple of refreshes on admin.ourdomain.com/health, which is definitely on a different IP address than the public subdomain, returns the health page for the public subdomain. That's bad.
On the bright side, I'm not seeing, for some reason, requests destined for the public subdomain's /health end up returning results from the admin side, but it's pretty disturbing anyway.
Whatever is going on, it might also be interesting to note that requests made on the wrong side, like admin.ourdomain.com/publicendpoint, are 404'd correctly. I'd imagine that's just because the /health is the only endpoint that inherently belongs to the API router, and moreover that that bolsters the case that whatever is happening seems to be happening because of an issue in the path from the GCE forwarding rule to the correct kubernetes service.
So, I guess we finally get to the part where I ask a question. Here goes:
Why are requests through an external ip associated with a forwarding rule targeting a particular kubernetes service being sent, intermittently, to the wrong kubernetes service?
Any assistance or information on this issue would be greatly appreciated.