3

I have multiple instances in an Amazon CloudFormation template and am trying to connect them together via the UserData, i.e. tell each other the IP Address of the other machine.

The template looks something like this:

"Instance1" : {
  "Type" : "AWS::EC2::Instance",
    ...
    "UserData" : { 
        "Fn::Base64" : { 
        "Fn::Join" : [ "\n", [ 
            { "Fn::Join" : [ "=", [ "Instance2", { "Fn::GetAtt" : [ "Instance2" , "PrivateIp"] } ] ] }
            ] ]
             } }
  }
},
"Instance2" : {
  "Type" : "AWS::EC2::Instance",
    ...
    "UserData" : { 
        "Fn::Base64" : { 
        "Fn::Join" : [ "\n", [ 
            { "Fn::Join" : [ "=", [ "Instance1", { "Fn::GetAtt" : [ "Instance1" , "PrivateIp"] } ] ] }
            ] ]
             } }
  }
},

Amazon CloudFormation refuses to handle this CloudFormation and reports that it cannot handle the circular dependency between the two instances.

Is there a way to work around this without having to build something on my own. I.e. I would like to have UserData on both instances that reflects the IP address of the other machine without chaninging the UserData manually afterwards.

1
  • 3
    You should be using a VPC for something like this. Then you can statically configure the private IP addresses and completely avoid this workaround.
    – jamieb
    Commented Mar 14, 2013 at 19:57

2 Answers 2

4

I actually figured out a way to do this with Cloud Formation templates alone by using ElasticIPs.

I create an ElasticIP (you cannot directly assign it to an instance in this case!)

"ServerEIP" : {
 "Type" : "AWS::EC2::EIP",
 "Properties" : {
 }
},

Then I reference that IP in the UserData

"Client" : {
   ...
        "UserData" : { 
            { "Fn::Join" : [ "=", [ "Server", { "Ref" : "ServerEIP" } ] ] }

The server can directly reference the client

"Server" : {
  "Type" : "AWS::EC2::Instance",
    ...
"UserData" : { 
        ...
            { "Fn::Join" : [ "=", [ "Client", { "Fn::GetAtt" : [ "Client" , "PrivateIp"] } ] ] },

And later on I associate the Elastic with the actual server to make Cloud Formation handle the dependencies correctly:

"ServerIPAssoc" : {
     "Type" : "AWS::EC2::EIPAssociation",
     "Properties" : {
         "InstanceId" : { "Ref" : "Server" },
         "EIP" : { "Ref" : "ServerEIP" }
     }
 },

Done! I now have two instances which know about the IP address of the other node.

The only downside is that the traffic now is done over a public ip address, thus incurring traffic costs and being probably less secure.

Update: I now run into the problem described here, not sure if I can work around this problem here.

1

You can accomplish your high level goal, but not with the constraints you have listed (i.e., including the raw IP addresses in user data for both instances). The simple reason is:

  • The user data must be specified before the instance is started.

  • The IP address is not known until after the instance is started.

CloudFormation can start one instance and provide its IP address to the second instance, but not both at once (circular dependency).

There are a number of approaches you could take and technologies you could use to solve this two way communication. At a high level:

  • You could pass the IP address of A into B, then have B contact A and let it know what it's IP address is (be careful with security).

  • You could store each instance's IP address in external storage (e.g., Route53, SimpleDB) and then each would query that external storage on startup to find their partner.

RECOMMENDATION

Here's a simple approach that can be used with CloudFormation and reliable AWS services:

  1. Set up a domain (hosted zone) in Route53. This could be completely separate from any public domain you currently use, though if you are already using Route53 you could plug this functionality into the same domain.

  2. In your CloudFormation template, generate a unique name for each instance, perhaps based on the current CloudFormation stack name (e.g., "MYSTACK-server-a.example.com" and "MYSTACK-server-b.example.com")

  3. In the CloudFormation template, pass the instance names into each server in their respective user-data.

  4. Add directives to your CloudFormation template to inject these new DNS names (record sets) into the Route53, mapping them to the IP addresses of the instances.

CloudFormation will start the instances, passing in the user data. When IP addresses have been assigned to the instances, CloudFormation will map them to the hostnames in Route53 DNS. Your instances can then use the hostnames to find each other.

If your instances need to find their partners at startup time, they will need to keep polling DNS until the partner has moved to the running state and been assigned an IP address. Be careful of using software where a DNS "miss" is cached.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .