I have an AWS Elastic beanstalk environment which can scale from 2 to 3 instances configured with an application load balancer. The load balancer has an HTTP healthcheck configured looking for a 200 response.
When the environment auto scales to 3 instances the new instance starts receiving traffic before it is ready. If I manually check the health check URL I can see that 1 in 3 times it returns the 404 because the new instance isn't ready. The other URLs of the app error too, 1 in 3 times because they don't exist.
As I understand it, the entire point of the health check URL is to deal with this. So, what could be causing the issue, please?
Some pieces of information that might be relevant:
- The instance takes a long time to become ready because it is an ASP.Net app that needs ARR installed.
- In the enhanced health check overview, the two working instances are listed as "Severe" even though they are working correctly. They have a couple of 5xx errors listed, I'm not sure why, and because little traffic is being sent to them it makes up a high percentage of the traffic. Bizarrely, the new instance that is not ready is marked as "Ok". I don't think enhanced health check dictates when the load-balancer considers an instance healthy, but perhaps I'm incorrect?
- When I tried to contact AWS support the agent sent me a load of information about immutable environment updates and rolling updates. As I understand it, this is relates to how new instances are brought oline when configuration changes or new deployments are made and this is different to the load-balancing health checks and auto-scaling, so not relevant. But, perhaps I misunderstood, and the agent didn't manage to communicate why this is relevant to this situation.
host
header to the instance's private IP address and I've seen people fail to take that into account. If all instances are failing their checks then ALB assumes the safest thing to do is forward traffic to all instances, which would explain what you see.