0

My Scenario

In a CloudFormation template I have a single VPC, a public subnet and a private subnet. In the public subnet I have amazons NAT AMI in an instance. In the private subnet I have an autoscaling group behind an internal LoadBalancer. This autoScaling group has a LaunchConfig to install httpd with a demo web page.

The problem

The EC2 instance launched in this private subnet autoscaling group does not install the webserver. This causes my ELB to fail and rollback the whole cloudformation stack. However, I can SSH in after creation, where I can successfully wget internet web pages and use yum install httpd manually. This fixes my cloudFormation stack by making the ELB check happy. /var/log/cloudinit-output.log says the instance couldn't resolve the amazon yum repository during its initialization.

I have a feeling this may be caused by the LaunchConfig being kicked off in a new EC2 instance before the NAT instance is fully up and working. I have tried adding 'DependsOn' : 'NATInstance' to the AutoScaling group but this has not fixed the issue.

Can you help?

2 Answers 2

1

Cloudwatcher was correct with his answer, however I would like to elaborate for anyone else who has a similar problem in the future.

The 'DependsOn' attribute in a CloudFormation template is met when a resource signals it is done. By default I believe this is when amazon has created the resource. In my example, the NAT instance had actually been created, which is when the instance was signaling. However, the config and settings inside the instance had not been completed, so the NAT remained non operational before other instances were trying to make use of it. The other instances then failed because they couldn't get an internet connection through the NAT instance.

You can manually override the default signalling yourself. THis means you can do your actions and THEN signal once its done. The 'DependsOn' attribute for all other resources relying on it will then work properly. You do this by using some amazon helper scripts inside an EC2 instance, specifically 'cfn-init' and 'cfn-signal'. In your 'UserData' property for an EC2 instance (or an auto-scaling group) you yum install aws-cfn-bootstrap to get the scripts (or which ever package manager you are using). You can then perform your initialization steps inside UserData and once complete, then signal the resources is done using cfn-signal. Here is my example:

"UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
            "#!/bin/bash -xe\n",
            "yum update -y aws-cfn-bootstrap\n",
            "wget <<URL FOR YOUR INIT BASH SCRIPT HERE>> -O - | bash\n",

            "/opt/aws/bin/cfn-init -v ",
            "         --stack ", { "Ref" : "AWS::StackName" },
            "         --resource <RESOURCE TO SIGNAL HERE> ",
            "         --region ", { "Ref" : "AWS::Region" }, "\n",

            "/opt/aws/bin/cfn-signal -e $? ",
            "         --stack ", { "Ref" : "AWS::StackName" },
            "         --resource <RESOURCE TO SIGNAL HERE> ",
            "         --region ", { "Ref" : "AWS::Region" }, "\n"
            ]]}}

I hope this helps someone.

1

There are several things to consider around security groups and such allowing traffic. But regarding the NAT specificly, make sure that in the NAT launch config you are not issuing the

/opt/aws/bin/cfn-signal

until your setup and passthrough scripts are complete. Given you are "DependsOn" the NAT it will not continue until the CloudFormation stack receives this signal.

[EDIT] If anyone is looking at this after today(2015-12-18) you should really consider moving the the NAT managed service provided by AWS. https://aws.amazon.com/about-aws/whats-new/2015/12/introducing-amazon-vpc-nat-gateway-a-managed-nat-service/

2
  • How do I go about only issuing this signal until setup is complete?
    – Toby
    Oct 24, 2015 at 21:36
  • 1
    In your LaunchConfig run through all your setup scripts and network forward logic. then make /opt/aws/bin/cfn-signal [NAT Instance Name] the last command in the script. Make sure that you are not running it until you other setup is complete Nov 6, 2015 at 20:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .