I am trying to autoscale my present instance, I am running a medium instance right now and auto scale with small instance. I used command line tool to config the settings, this is the configurations I have used to scale and I am running minimum of one instance apart from my regular instance, that means one more instance and I have attached to the load balancer.

s-create-auto-scaling-group groupname --launch-configuration launchconfig --availability-zones ap-southeast-1a --min-size 1 --max-size 5 --load-balancers prod

But when I have checked the load balancer it says "Out of Service" reason "Instance has failed at least the Unhealthy Threshold number of health checks consecutively". How can I solve this, using its public DNS, I am not able to get any response from the instance and also not able to ssh to it since the key value pair is not attached to the newly created instance.

what is the problem.How do I solve this.

Please help me out its little urgent, since I am struck up at this point from almost 2 days.

as-describe-launch-configs --show-long --headers


as-describe-auto-scaling-groups --show-long --headers


i-43796716  testASG  ap-southeast-1a  InService  HEALTHY  testLC

elb-describe-lbs --headers --show-long

prod,prod-11719395.ap-southeast-1.elb.amazonaws.com,prod-11719395.ap-southeast-1.elb.amazonaws.com,Z1WI8VXHPB1R38,"{interval=120,target=HTTP:80/user/sign_in/,timeout=30,healthy-threshold=5,unhealthy-threshold=3}",ap-southeast-1a,(nil),(nil),"i-495dda1c, i-43796716","{protocol=HTTP,lb-port=80,instance-protocol=HTTP,instance-port=80,policies=AWSConsolePolicy-1}",(nil),"{policy-name=AWSConsolePolicy-1,expiration-period=180}","{owner-alias=amazon-elb,group-name=amazon-elb-sg}",(nil),2012-02-01T10:36:08.810Z

elb-describe-instance-health loadbalancername --headers --show-long

INSTANCE_ID,i-43796716,OutOfService,Instance has failed at least the UnhealthyThreshold number of health checks consecutively.,Instance
  • 1
    Please provide relevant portions of the output from the following: as-describe-launch-configs --show-long --headers, as-describe-auto-scaling-groups --show-long --headers, as-describe-auto-scaling-instances,elb-describe-lbs --headers --show-long, elb-describe-instance-health loadbalancername --headers --show-long. Also (to confirm that the necessary ports are open on your autoscaled instances): ec2-describe-group. (Without more info - I'd guess a security group issue on your autoscaled instance; but there are quite a few other possibilities). – cyberx86 Feb 3 '12 at 1:12
  • 1
    I am still not quite clear on the 'both instances belong to the same group' part - is the small instance the only instance being created by autoscaling? Meaning that the medium instance is independent (although, behind the same load balancer). How are you verifying that DNS is not responding? I presume that DNS specific tasks (e.g. dig should work (if not, please add that to your question) - but the security group controls everything else (including ping - so if incorrectly set, the instance will be completely inaccessible - which sounds like your scenario). Does it work without the LB? – cyberx86 Feb 3 '12 at 5:21
  • 1
    Also - the requested configs, above, are not going to fit in a comment - so just add them to your question. (Given that it is 12:30 am here, I won't be able to look at them for quite a few hours - but they will help anyone else who looks at the question.) If you are testing with ping - see this to permit it through your security group - and check for other restrictions (e.g. only permitting traffic from a particular source). Finally see, if you can relaunch your m1.small with a key so you can ssh in. – cyberx86 Feb 3 '12 at 5:32
  • 1
    One more point of mention - check your console log on the m1.small if you still can't reach it - it is possible your instance didn't start correctly for some reason (e.g. unable to mount a disk - shouldn't be a problem with a c1.medium to m1.small, but would be a problem for a m1.large to m1.small); (Also, you will, of course, need to setup the security group on the autoscaled instance to allow SSH, even once you have the key in place - out of curiosity, what happened when you tried to SSH into the non-functional m1.small that didn't have a key? Did it connect before giving an error? – cyberx86 Feb 3 '12 at 6:29
  • 1
    both security group and keys need to be set as part of the launch config. Using as-create-launch-config, pass the --key YOUR_KEY_NAME and --group YOUR_SECURITY_GROUP. The permission denied, public key error suggests that you are able to connect to the instance, but not authenticate (since, no key). Will take a look at the configs in a few hours - have a class to teach in a couple minutes. – cyberx86 Feb 3 '12 at 16:43

There are a number of considerations to take into account here. Firstly, to resolve the most limiting problem - the lack of SSH access.

Since your previous launch config did not specify a keypair, you will have no valid credentials with which to access the instance. Unfortunately, the initial keypair cannot be added after the instance is launched.

To remedy this, you must create a new launch config, passing the --key and --group parameters, in addition to all the parameters you passed previously. --key takes the name of the keypair you want to use, while --group takes the security group name (if not in VPC) or ID.

In cases where you can't access your instance, the console log can help you to verify the instance has in fact successfully booted. A common issue is boot failure due to missing volumes (especially trying to mount ephemeral volumes that exist only on the larger instance types, when booting a smaller instance type).

An important point of mention is that an AMI is not updated if you change a running instance. You must explicitly create the new image. As such, if you try to launch a new instance using the same AMI you are currently using on a customized instance, there is a good chance that you will simply be launching one of the default AMIs, and not one with your customizations on it.

Use ec2-describe-images to determine the block device mapping of your image - and the snapshot the volume is based on - this will verify that you will be mounting an EBS volume that has your customizations built into it.

If you do not have an up to date AMI to use for autoscaling:

  • Create a snapshot of your EBS volume
  • Create an AMI with ec2-register -n IMAGE_NAME -s SNAPSHOT_ID
    • If you have additional EBS volumes to attach, specify those by adding the --block-device-mapping (-b) parameter (e.g. -b /dev/xvdf=SNAP_ID)
  • Verify that you have the correct block device mappings with ec2-describe-images

Once you have an up to date AMI, you need to create a new launch config that will use this AMI. If desired, you can pass additional block device mappings to the command. Use as-create-launch-config, passing it your new AMI and all the parameters you used previously.

Finally, you must update your autoscaling group. This group is associated with a particular launch config - the new launch config will not be automatically detected and has no effect on the autoscaling group until you explicitly associate it. Use as-update-auto-scaling-group GROUP_NAME --launch-configuration CONFIG_NAME to make this change.

Once the changes have been made, you can simulate an autoscaling event using the as-execute-policy command.

Remember to give your instances a few minutes to boot up - if your ELB is showing instances as unhealthy, you may want to increase the --grace-period of your autoscaling group.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.