How do you determine reasonable health check timeouts for load balancers?
My application is failing load balancer health checks. I'm using the default health check timeout of 5 seconds, but I've noticed that average latency graphs on CloudWatch jump up to ~50s during periods (lasting between 2-4 hours) when the application runs at its peak of ~30% CPU utilization. Memory utilization and IOPs are all low and stable. Is a 30% utilization high enough to expect health check responses to increase beyond 5 seconds? If so, is there a standard practice of determining the health check timeout?