3

I have an AWS Elastic beanstalk environment which can scale from 2 to 3 instances configured with an application load balancer. The load balancer has an HTTP healthcheck configured looking for a 200 response.

When the environment auto scales to 3 instances the new instance starts receiving traffic before it is ready. If I manually check the health check URL I can see that 1 in 3 times it returns the 404 because the new instance isn't ready. The other URLs of the app error too, 1 in 3 times because they don't exist.

As I understand it, the entire point of the health check URL is to deal with this. So, what could be causing the issue, please?

Some pieces of information that might be relevant:

  • The instance takes a long time to become ready because it is an ASP.Net app that needs ARR installed.
  • In the enhanced health check overview, the two working instances are listed as "Severe" even though they are working correctly. They have a couple of 5xx errors listed, I'm not sure why, and because little traffic is being sent to them it makes up a high percentage of the traffic. Bizarrely, the new instance that is not ready is marked as "Ok". I don't think enhanced health check dictates when the load-balancer considers an instance healthy, but perhaps I'm incorrect?
  • When I tried to contact AWS support the agent sent me a load of information about immutable environment updates and rolling updates. As I understand it, this is relates to how new instances are brought oline when configuration changes or new deployments are made and this is different to the load-balancing health checks and auto-scaling, so not relevant. But, perhaps I misunderstood, and the agent didn't manage to communicate why this is relevant to this situation.
4
  • 2
    Does your health check actually work at all, ever when the balancer is doing the checking? That's not necessarily the same thing as when you manually check it, which you say is working. The balancer will set the HTTP host header to the instance's private IP address and I've seen people fail to take that into account. If all instances are failing their checks then ALB assumes the safest thing to do is forward traffic to all instances, which would explain what you see. – Michael - sqlbot Apr 24 '20 at 1:16
  • Thanks @Michael-sqlbot that was it. I checked the logs, which I should have done before, and the URL Check was failing for all instances because the load balancer was using the IP rather than the domain that the current health check relied upon behind the scenes. I should have checked the log before; I blame it on having been at the end of a long day and lock-down meaning I don't have a colleague next to me to call over and check for silly oversights ;-) If you want to leave a formal answer then I'll mark it. – Steve Apr 24 '20 at 9:13
  • @Michael-sqlbot Make that an answer? – Hauke Laging Apr 25 '20 at 13:57
  • @HaukeLaging thanks for the reminder. Done. – Michael - sqlbot Apr 25 '20 at 14:33
2

If I manually check the health check URL I can see that 1 in 3 times it returns the 404 because the new instance isn't ready.

The fact that the health check URL works against the healthy nodes when you happen access one of them through the balancer is not necessarily the same thing as the health check URL working correctly when the balancer accesses it to do a health check.

Modern HTTP versions require that a Host header be present in each incoming request, and the balancer will set the HTTP host header to the instance's private IP address for health check requests, but will pass through the value set by the browser on normal requests -- and your browser is setting that same header to the hostname you're using to access the balancer.

If you (and your servers/framework/application) fail to take that into account, and your server treats them differently, then all of your instances may actually be failing their health checks all the time despite manual checks working when you try them. When that "all targets unhealthy" condition happens, ALB assumes the safest thing to do is forward traffic to all instances, as if they were all healthy (a fail-safe but not necessarily intuitive design) which would exactly explain what you see happening.

If a target group contains only unhealthy registered targets, the load balancer nodes route requests across its unhealthy targets.

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.