0

We're using GCE and getting a very odd error:

"Error loading health status"

We see this error on all of our HTTP load balancers. The health check failure doesn't seem to be having any impact on the actual service. In other words, the server itself works just fine, but the health check is "red."

We use DM to setup our environment, so what we have running now is exactly the same as it's always been. Up until today we have been using the beta API's for everything. Our theory was that if we moved everything over to using "v1" we would resolve this issue. What we learned is that even with all "v1" bits, we still see the same error.

tcpdump -vvvs 1500 -l -A port 80

20:01:21.249194 IP (tos 0x80, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 146)
    130.211.1.239.42560 > master-game-100-tbmatch-us-central1-a-xxxx.c.radiant-cloud.internal.http: Flags [P.], cksum 0xe9bd (correct), seq 1:95, ack 1, win 222, options [nop,nop,TS val 92585061 ecr 407465], length 94
E.....@.@.......
[email protected]...............
...e..7.GET /healthz HTTP/1.1
Host: 10.240.0.30
User-Agent: GoogleHC/1.0
Connection: Keep-alive

We see the health check request coming in...

20:01:21.250109 IP (tos 0x0, ttl 63, id 15146, offset 0, flags [DF], proto TCP (6), length 155)
    master-game-100-tbmatch-us-central1-a-0buf.c.radiant-cloud.internal.http > 130.211.1.239.42560: Flags [P.], cksum 0x905d (incorrect -> 0xb6e0), seq 1:104, ack 95, win 220, options [nop,nop,TS val 407465 ecr 92585061], length 103
E...;*@.?.pc
[email protected]......].....
..7....eHTTP/1.1 200 OK
Content-Length: 2
Content-Type: text/plain
Date: Mon, 11 Jan 2016 20:01:21 GMT

ok

And we see our service responding appropriately like it always has in the past. Is anyone else seeing this problem?

5
  • I'm voting to close this question as off-topic because this is an infrastructure problem ad as such should be referred to Google.
    – user9517
    Jan 11, 2016 at 20:25
  • We are currently unable to refer this to google as our support contact has expired. We recently graduated from the startup program ( w00t ) which means we no longer have gold support ( boo ). Jan 11, 2016 at 20:51
  • We can't help with the google infrastructure only they can.
    – user9517
    Jan 11, 2016 at 20:55
  • Agreed, and I'm not disagreeing with you here, or trying to be argumentative. My goal here was to find out if anyone else is seeing this same problem, of if it's just us. Jan 11, 2016 at 20:57
  • That's not what we do here. This is a Q&A site, not a forum. Ask on the relevant Google group is you don't have a contract.
    – user9517
    Jan 11, 2016 at 20:59

1 Answer 1

1

There are few things you need to check:

  1. Check the iptables on your instance to confirm if they are blocking the traffic from IP 169.254.169.254 (this IP belongs to the metadata server and the health checks are sent to the instance from this IP).

  2. Run a tcpdump from any instance behind the load balancer by running the command: tcpdump -­A -­n host 169.254.169.254 and check the flags

  3. Check if "google­address­manager" is running on your instance by using the command ps aux | grep google­address*. Google­address­manager is a startup script responsible for managing all the addresses from the project metadata to the instances.

  4. Check if the health check is listening on the same port as the service running on the instance.

  5. I would recommend setting the "Unhealthy threshold" parameter to 5 instead of the default 2, if your setting has the default value.

More information can be found here

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .