Can't create EC2 instance with Keypairs for ssh access

Question

I am trying to launch a new EC2 instance with docker installed on it on AWS. Later on I need to ssh into this instance. I can create it without a problem doing:

docker-machine create --driver amazonec2 --amazonec2-region=eu-central-1 machine-ec2

This creates a new machine and also a keypair. Problem is it doesn't download the .pem file I need later on to ssh into it.

Thus, I tried to create a new machine with an existing keypair. I created a keypair, then I downloaded it, copied it to my ./ssh/aws folder and ran:

docker-machine create --driver amazonec2 --amazonec2-keypair-name=machine-aws-keypair --amazonec2-ssh-keypath=~/.ssh/aws/ --amazonec2-region=eu-central-1 machine-ec2

that gives me the error:

Error creating machine: Error in driver during machine creation: unable to create key pair: open ~/.ssh/aws/: no such file or directory
equally: docker-machine create --driver amazonec2 --amazonec2-keypair-name=machine-aws-keypair --amazonec2-ssh-keypath=~/.ssh/aws/nameofmykeyfile.pem --amazonec2-region=eu-central-1 machine-ec2

Is there sth I am doing wrong or is this a problem of AWS? I think the former... Any ideas on how to fix this? Help is very much appreciated. Thanks in advance!

EDIT: It would be enough to know how to get my privatekey.pem when creating a new instance via terminal....

Answer 1

I created a keypair, then I downloaded it, copied it to my ./ssh/aws folder and ran:

This path means a folder called ssh under . (current directory)

unable to create key pair: open ~/.ssh/aws/: no such file or directory

This path means a folder called .ssh under ~ (your home)
Try putting the keys in ~/.ssh/aws/ and see if that helps.

Answer 2

If the key pair does not exist (in the AWS region required), it can be created using the aws-cli

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/create-key-pair.html

For example, this assumes your AWS credentials are enabled already and your IAM permissions allow creating a key pair:

KEY_NAME=your-unique-name

export AWS_DEFAULT_REGION=us-west-2
export AWS_KEYPAIR_NAME="${KEY_NAME}-${AWS_DEFAULT_REGION}"
export AWS_SSH_KEYPATH="${HOME}/.aws"

aws ec2 create-key-pair \
    --key-name "${AWS_KEYPAIR_NAME}" \
    --query "KeyMaterial" \
    --output text > \
    "${AWS_SSH_KEYPATH}"/"${AWS_KEYPAIR_NAME}.pem"

chmod 400 "${AWS_SSH_KEYPATH}"/"${AWS_KEYPAIR_NAME}.pem"

ssh-keygen -y -f \
    "${AWS_SSH_KEYPATH}"/"${AWS_KEYPAIR_NAME}.pem" > \
    "${AWS_SSH_KEYPATH}"/"${AWS_KEYPAIR_NAME}.pub"

However, perhaps the best practice is to allow docker-machine to manage the ssh keys rather than trying to instruct it to use a specific key. This isolates access to each machine in a more secure manner and relieves the user of managing ssh keys.

One advantage to a common ssh key is the ability to use parallel ssh. A simple substitute for this is a bash loop. (I don't know about solutions for capistrano, for example.) For example, assuming several machines have a common machine-name prefix and a numeric suffix:

machine_prefix="my-machines"
for n in $(seq 1 6);
do
  docker-machine ssh "${machine_prefix}-00$n" 'echo $(hostname)'
done

See https://github.com/dazza-codes/docker-machine-ec2/blob/master/ec2_spinup.sh#L87-L94 for an example of creating machines with a common machine name prefix (assuming docker-machine is the solution of choice, vs. docker swarm, AWS Batch, AWS Labmda, k8s, etc).

Answer 3

Leaving this for future readers who struggle with the same problem:

So what AWS actually does when creating an instance with the command line is that it creates a keypair and downloads it to your local machine.

To find out where it is located one can:

List all docker-machines docker-machine ls Inspect the machine of the EC2 instance docker-machine inspect machinename

This will give you sth like:

{
    "ConfigVersion": 3,
    "Driver": {
        "IPAddress": "3.124.195.229",
        "MachineName": "machine-ec2",
        "SSHUser": "xxx",
        "SSHPort": xxx,
        "SSHKeyPath": "pathtokey/id_rsa",
        "StorePath": "/Users/username/.docker/machine",
        ....


}

To ssh into the the EC2 instance do:

ssh -i "pathtokey/id_rsa" ubuntu@endpointofyourec2instance

And you should be in.