We run some web server on AWS in an small auto-scaling group (2-4 instances usually). Its healthcheck is based on the ELB healthcheck such that if one of the web servers stops responding it will be replaced by a new instance.
This is currently working well. If there is an outage the non-responsive servers are terminated and new ones are spun up to take their place.
The problem is that we currently have no idea why the servers are going down. We have some suspicions from some of the logs we have but we can no longer log onto the server, look through the web logs, the system logs or anything else.
I know that there is a way to manually detach instances from an auto scaling group but is there any way to do this automatically in the event of a failure?
That is what I would like to happen is that when a server fails it is left running but removed from the auto-scaling group so that we can then analyze it to see what went wrong. An obvious bonus would be to limit how often this might occur so that if for some reason it keeps cycling servers we don't end up with hundreds of detached instances.
So can this be done somehow and if so how?