Knife can't ssh into newly instantiated EC2 server

Question

I just set up a new Chef environment as I'm currently expanding my knowledge of Chef. I have a Key Pair set up on EC2, I have my Knife configuration set up. When I attempt to spawn a server, the node is created but Knife can't ssh into it.

Here's my knife.rb (which is outside of the repo):

current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name                "mynode"
client_key               "/Users/me/.chef/my.pem"
validation_client_name   "my-validator"
validation_key           "/Users/me/.chef/my-validator.pem"
chef_server_url          "https://api.opscode.com/organizations/myorg"
cache_type               'BasicFile'
cache_options( :path => "/Users/me/.chef/checksums" )
cookbook_path            ["/Users/me/git/chef/cookbooks"]

knife[:aws_access_key_id] = "yadayadyada"
knife[:aws_secret_access_key] = "blahblahblah"
knife[:identity_file] = "/Users/me/.ssh/knife.pem"
knife[:aws_ssh_key_id] = "knife"

Here's my knife command:

knife ec2 server create -r "role[whatever]" -I ami-09470539 --subnet subnet-03e44866 -f t2.micro --ssh-user ubuntu --region us-west-2 -Z us-west-2a

I also tried it by specifying the pem directly:

knife ec2 server create -r "role[whatever]" -I ami-09470539 --subnet subnet-03e44866 -f t2.micro -S knife -i ~/.ssh/knife.pem --ssh-user ubuntu --region us-west-2 -Z us-west-2a

This is a HVM instance inside a VPC group.

What I've tried and checked...

1. Yes, the pem has the right permissions (400).
2. Yes, the EC2 security group ("default") is world-accessible on port 22.
3. Yes, I can ssh into it directly using the knife.pem key on the command line.
4. Yes, I've Googled this exhaustively and read three different tutorials. I seem to have done everything correctly.

Is there anything else that I'm missing?

In verbose mode, this is what I am seeing...

Waiting for sshd
.DEBUG: ssh timed out: 172.nnn.nnn.nnn
.DEBUG: ssh timed out: 172.nnn.nnn.nnn

Answer 1

I ran into this exact problem when I was first setting up chef on EC2. Here's the command we use to launch EC2 instances with knife successfully:

knife ec2 server create \
--flavor m3.medium \
--image ami-****** \
--iam-profile "iam-app" \
--ebs-size 30 \
--security-group-ids sg-**** \
--subnet subnet-6de**** \
--ssh-key my-key-name \
--ssh-user ubuntu \
--ssh-port 22 \
--identity-file "/local/path/to/ssh/key/for/this/instance" \
--ssh-gateway [email protected] \ #remove this line if you're not connecting through a bastion host
--server-connect-attribute private_ip_address \ # Because we connect through a bastion host we want to explicitly connect to the the private IP address.  You may want to set this to the public IP address.  I believe these are fog attributes.
--node-name "test-play-1" \
--tags Name="test-play-1",Environment="Test" \
--run-list "role[app]" \
--environment test

Note that it's best practice to use a Bastion Host to connect to your instances versus connecting directly to each EC2 instance. Also, for public-facing servers, we use a line like this to explicitly assign the Elastic IP Address:

--associate-eip 54.186.***.*** 

Answer 2

Just wanted to add what worked for us without a bastion host since it took a lot of experimenting:

knife ec2 server create   --image ami-xxxxx                       \
                      --flavor t2.medium                      \
                      --run-list 'recipe[recipe-name]'        \
                      --security-group-id sg-xxxxx            \
                      --region us-east-1                      \
                      --node-name $1 -T Name=$1               \
                      --subnet subnet-xxxxx                   \
                      --ssh-user ubuntu                       \
                      --ssh-key key_name                      \
                      --identity-file "~/.ssh/key_file_name"  \
                      --server-connect-attribute dns_name     \
                      --associate-eip $2                      \
                      --associate-public-ip                   \
                      --no-host-key-verify      

where the first parameter is the instance name and the second parameter is the public elastic IP.

Cheers!

Answer 3

Well of course, as soon as I walked away from the computer, I realized what the issue is...

knife is trying to ssh in using the private IP. D'oh!

I can only assume that --associate-public-ip will fix that. I'm away from the computer and will test later. Feel free to confirm/deny/offer advice.

Answer 4

Since I am connecting over public IP and this --server-connect-attribute public_ip_address works for me.