Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

i'm trying to define a cluster of Cassandra nodes, together with some other instances ( some middleware and app servers ). I'd like to be able to add the ( internal ) IP addresses of the Cassandra nodes to the configuration of the middleware and also add the IP of the first Cassandra node to the configuration of the other nodes, as a seed node. Can i achieve this entirely from within the template ? ( invoking scripts in the AMI is ok, centralized configuration or any kind of remote database not provided by the platform is not )

thanks! Andras

edit: i forgot to mention i would also need to be updating the relevant security groups with the IPs of the instances.

share|improve this question
add comment

3 Answers

up vote 4 down vote accepted

After some digging in the AWS docs, i've found that you can reference the properties of resources created in the same template, as long as they a) are defined before your reference b) don't point to the resource being created ( no circular dependencies )

b) implies that to find out an EC2 node's own IP i indeed have to call the aws URL ceejayoz mentions above

This is my definition of an Instance resource for a Cassandra node. Note that i have placed markers in the configuration file to be able to safely insert correct values, but you could always revert to simply replacing the default values ( e.g. "initial_token: 0" with "initial_token: 123" )

    "Cas1" : {

        "Type" : "AWS::EC2::Instance",
        "Metadata" : {
            "AWS::CloudFormation::Init" : {
                "config" : {
                    "packages" : {
                        "yum" : {
                            "opscenter-free" : [],
                            "pyOpenSSL" : []
                        }
                    }
                },

                "services" : {
                    "sysvinit" : {
                        "cassandra" : {
                            "enabled" : "true",
                            "ensureRunning" : "true"
                        },
                        "opscenterd" : {
                            "enabled" : "true",
                            "ensureRunning" : "true"
                        }
                    }
                }
            }
        },

        "Properties" : {
            "SecurityGroups" : [ { "Ref" : "CommonSecurityGroup" }, { "Ref" : "OpsCenterSecurityGroup" } ],
            "KeyName" : { "Ref" : "KeyName" },
            "InstanceType" : { "Ref" : "CassandraInstanceType" },
            "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },{ "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "CassandraInstanceType" }, "Arch" ] } ] },
            "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
            "#!/bin/bash -v\n",

            "MY_IP=`(curl http://169.254.169.254/latest/meta-data/local-ipv4)`\n",
            "sed -i 's/REPLACE_WITH_TOKEN/0/g' /etc/cassandra/conf/cassandra.yaml\n",
            "sed -i 's/REPLACE_WITH_MY_IP/'$MY_IP'/g' /etc/cassandra/conf/cassandra.yaml\n",
            "sed -i 's/REPLACE_WITH_SEED_IP/'$MY_IP'/g' /etc/cassandra/conf/cassandra.yaml\n",
            "sed -i 's/REPLACE_WITH_MAX_HEAP_SIZE/8G/g' /etc/cassandra/conf/cassandra-env.sh\n",
            "sed -i 's/REPLACE_WITH_HEAP_NEWSIZE/4G/g' /etc/cassandra/conf/cassandra-env.sh\n",

            "yum update -y aws-cfn-bootstrap\n",

            "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" }, " -r Cas1 ",
            "    --access-key ",  { "Ref" : "HostKeys" },
            "    --secret-key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},
            "    --region ", { "Ref" : "AWS::Region" }, " || error_exit 'Failed to run cfn-init'\n",

            "sed -i 's/127.0.0.1/'$MY_IP'/g' /etc/opscenter/opscenterd.conf\n",
            "sed -i 's/#passwd_file/passwd_file/g' /etc/opscenter/opscenterd.conf\n",
            "echo 'xxx:xxx:admin' > /etc/opscenter/.passwd\n",

            "rm -fR /var/lib/cassandra/data\n",
            "rm -fR /var/lib/cassandra/commitlog\n",
            "mkdir /var/lib/cassandra/data\n",
            "mkdir /var/lib/cassandra/commitlog\n",
            "chown -R cassandra:cassandra /var/lib/cassandra\n",

            "service cassandra start\n",
            "service opscenterd start\n"

            ]]}}
        }

    },

Configuration of subsequent nodes are the same, except the line

            "sed -i 's/REPLACE_WITH_SEED_IP/'$MY_IP'/g' /etc/cassandra/conf/cassandra.yaml\n",

References the IP address of the seed ( first ) node

            "sed -i 's/REPLACE_WITH_SEED_IP/", { "Fn::GetAtt" : ["Cas1", "PrivateIp"] }, "/g' /etc/cassandra/conf/cassandra.yaml\n",

With regards to adding the instance IPs to security groups, this seems unnecessary, you can just create a group and add Ingress resources to it in which you reference the group as the Source

    "CommonSecurityGroup" : {
        "Type" : "AWS::EC2::SecurityGroup",
        "Properties" : {
            "GroupDescription" : "Enables",
            "SecurityGroupIngress" : [ {
                "IpProtocol" : "tcp",
                "FromPort" : "22",
                "ToPort" : "22",
                "CidrIp" : "0.0.0.0/0"
            } ]
        }
    },

    "OpsCenterSecurityGroup" : {
        "Type" : "AWS::EC2::SecurityGroup",
        "Properties" : {
            "GroupDescription" : "Enables",
            "SecurityGroupIngress" : [ {
                "IpProtocol" : "tcp",
                "FromPort" : "8888",
                "ToPort" : "8888",
                "CidrIp" : "0.0.0.0/0"
            },
            "SecurityGroupIngress" : [ {
                "IpProtocol" : "tcp",
                "FromPort" : "61620",
                "ToPort" : "61621",
                "SourceSecurityGroupName" : { "Ref": "CommonSecurityGroup" }
            } ]
        }
    },

    "CassandraThriftIngress" : {
        "Type" : "AWS::EC2::SecurityGroupIngress",
        "Properties" : {
            "GroupName": { "Ref": "CommonSecurityGroup" },
            "IpProtocol" : "tcp",
            "FromPort" : "9160",
            "ToPort" : "9160",
            "SourceSecurityGroupName" : { "Ref": "CommonSecurityGroup" }
        }
    },

    "CassandraData" : {
        "Type" : "AWS::EC2::SecurityGroupIngress",
        "Properties" : {
            "GroupName": { "Ref": "CommonSecurityGroup" },
            "IpProtocol" : "tcp",
            "FromPort" : "7000",
            "ToPort" : "7001",
            "SourceSecurityGroupName" : { "Ref": "CommonSecurityGroup" }
        }
    },

    "CassandraJMX" : {
        "Type" : "AWS::EC2::SecurityGroupIngress",
        "Properties" : {
            "GroupName": { "Ref": "CommonSecurityGroup" },
            "IpProtocol" : "tcp",
            "FromPort" : "7199",
            "ToPort" : "7199",
            "SourceSecurityGroupName" : { "Ref": "OpsCenterSecurityGroup" }
        }
    },
share|improve this answer
add comment

Yes, you can do this entirely within the template. Below is the "Resources" section of a template which defines three instances: Two Cassandra instances, and one middleware instance.

The second Cassandra instance receives the IP of the first as user-data. The middleware instance receives the IPs of both Cassandra instances, also as user-data.

Both Cassandra instances are added to the 'cassandra-nodes' security group. I have not included the definition for this, but there is an example of a security group template in the snippets section. You do not need to manually add their IPs to the SG, just reference it when describing the instance.

"Resources" : {
    "CassOneInstance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "ImageId" : "ami-123",
        "InstanceType": "c1.medium",
        "SecurityGroups" : [ "cassandra-nodes" ]
      }
    },
    "CassTwoInstance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "ImageId" : "ami-123",
        "InstanceType": "c1.medium",
        "SecurityGroups" : [ "cassandra-nodes" ],
        "UserData" : { 
            "Fn::Base64" : { 
                "Fn::Join" : [ ":", [ 
                    { "Fn::GetAtt" : [ "CaseOneInstance", "PrivateIP" ] }
                ] 
            }
         },
      }
    },
    "MiddleWareInstance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "ImageId" : "ami-123",
        "InstanceType": "c1.medium",
        "UserData" : { 
            "Fn::Base64" : { 
                "Fn::Join" : [ ":", [ 
                    { "Fn::GetAtt" : [ "CaseOneInstance", "PrivateIP" ] },
                    { "Fn::GetAtt" : [ "CaseTwoInstance", "PrivateIP" ] }
                ]
            }
         },
      }
    }
}
share|improve this answer
add comment

invoking scripts in the AMI is ok

If so, all you need is cURL and EC2's internal metadata service. From the instance:

curl http://169.254.169.254/latest/meta-data/local-ipv4

There's a whole bunch of other available metadata at http://169.254.169.254/latest/meta-data/ - hostname, security groups, instance ID, etc.

share|improve this answer
    
Can i use this url to retrieve information on other instances ( i.e. "all the cassandra nodes in stack X" or "the first cassandra node in stack X" ? i was under the impression this would only give me info on the instance i'm sending the http requests from. thanks! –  pgn Apr 25 '12 at 22:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.