GCE Health Check settings are reverting to default

Question

I set up a Load Balancer on the GCP using an Ingress in the Kubernetes Engine.

The Backend Service by default registered all instances as unhealthy. I updated the health check to the HTTPS protocol and the /healthz url, since that's the url for my health check.

I leave it alone for a couple of minutes and the backend service states all instances are healthy now, then I see in the logs that my pod is serving 200 responses at the /healthz.

I navigate to the page, and it starts showing a 502 error. I go back to the health check page in the console, and the health check reverted to HTTP at /.

What caused the health check to revert back to HTTP? Here's the logs showing that for 10 mins the health check continues to request the https I set, then the request changes back to http.

...
[W 180621 20:02:35 iostream:1451] SSL Error on 9 ('10.128.0.14', 48346): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:02:35 iostream:1451] SSL Error on 9 ('10.128.0.13', 63030): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.14) 0.75ms
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.13) 0.80ms
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.14) 0.74ms
...
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.13) 1.26ms
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.14) 0.63ms
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.15) 0.64ms
[W 180621 20:05:46 iostream:1451] SSL Error on 9 ('10.128.0.14', 49971): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:46 iostream:1451] SSL Error on 9 ('10.128.0.15', 62893): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:48 iostream:1451] SSL Error on 9 ('10.128.0.13', 52191): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:48 iostream:1451] SSL Error on 9 ('10.128.0.13', 60549): [SSL: HTTP_REQUEST] http request (_ssl.c:833)

EDIT: I'm adding the readiness probe for the pod in question, that the ingress is attached to.

readinessProbe:
  httpGet:
    port: 8902
    scheme: HTTPS
    path: /healthz
  initialDelaySeconds: 5
  periodSeconds: 10
  successThreshold: 1

Answer 1

I checked and it shouds an intended behaviour.

Kubernetes controls the health check and configures it from data inside the kubernetes cluster. Kubernetes does not see any changes made to the health check via gcloud.

Therefore I guess it is normal that it checks it and any modification get reverted.

The solution to use readinessProbe instead.

---

Please take a look also to this:

Health checks

Currently, all service backends must satisfy either of the following requirements to pass the HTTP(S) health checks sent to it from the GCE loadbalancer:

1. Respond with a 200 on '/'. The content does not matter.

2. Expose an arbitrary URL as a readiness probe on the pods backing the Service.

The Ingress controller looks for a compatible readiness probe first, if it finds one, it adopts it as the GCE loadbalancer's HTTP(S) health check. If there's no readiness probe, or the readiness probe requires special HTTP headers, the Ingress controller points the GCE loadbalancer's HTTP health check at '/'.

Answer 2

I had the same issue.

The issue was that my readinessProbe was set to HTTPS only

readinessProbe:
  httpGet:
    path: "/health?ready=1"
    port: 8080
    scheme: HTTPS

However GCE Backends automatically assume that your services respond to HTTP and by default will scan for health checks there.

You can read more about GKE Ingresses and how Heath Checks respond HERE

To fix this I needed to update my service to respond with only HTTPS

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"my-port-name":"HTTPS"}'
spec:
  ports:
  - port: 8080
    targetPort: 8080
    name: my-port-name

This results in the backend service that the ingress creates to scan for readinessProbes over HTTPS.

NOTE: Be sure to tear down your Ingress, LoadBalancer, and HealthCheck before doing this then recreate it after as it will not respond to updates applied to your service.

Answer 3

Looks like your backends are serving only on HTTPS protocol but you've deployed an Ingress resource for HTTP protocol. Manually changing the configuration after deployment is not the solution as the master would revert the changes to comply with the manifest. Per this document you can secure an Ingress by specifying a secret that contains a TLS private key and certificate. Currently the Ingress only supports a single TLS port, 443, and assumes TLS termination.