I have an instance group with 2 instances behind a HTTP Load Balancer. one instance is up and functioning normally (returning http 200), the other is crashed (HTTP requests timeout). I am not sure what I'm doing wrong, but according to the documentation the failed instance should automatically be removed from the load balancer.
Here is the related docs: https://cloud.google.com/compute/docs/load-balancing/health-checks with the related paragraph:
For a health check to be deemed successful, the backend must return a valid HTTP response with code 200 and close the connection normally within the timeoutSec period. If an instance fails its health check, it is removed from the group or pool without any notification being sent. If it later passes a heath check, it is returned to the group or pool, again without any notification.
Here is what I currently see on my google cloud console page for the HTTP Load Balancer's Backend.
When visiting my site (http://stage2.phantomjscloud.com) Rougly half the time I get
Error: Server Error The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.
The HTTP Load Balancer (and health check) clearly detects the failed instance, but traffic is still being served to it regardless.
How can I resolve the issue?