1

I'm using a CloudFormation template to boot up a custom windows AMI, which needs to retrieve some code from an S3 bucket and execute it, once it boots up.

After quickly being able to do it manually using the AWS CLI (s3 sync) I've been struggling to get this to work via CloudFormation for over 8 hours now, to no avail. In essence, this command always fails with:

"NoCredentialsError: Unable to locate credentials"

Initially I tried setting the /.aws/credentials and /.aws/config files, in many different ways, until I finally realized that the user identity running cfn-init.exe and it's child processes doesn't have access to these files at all and that won't work. So instead I've opted for setting environment variables using setx, but that doesn't seem to work either and I'm still getting the same error.

I'm getting quite frustrated with this as every test requires a change to the CF template, uploading to S3, creating a stack, waiting a good 5-10 minutes for it to finish bootstrapping only to RDP and find out it failed, again.

Here's the init > config portion of my template:

"commands" : {
  "0-Tester" : {
    "command" : "echo \"I am OK.\" > \"d:\\test.txt\""
  },
  "1-SetAK" : {
    "command" : { "Fn::Join" : ["", [
      "setx AWS_ACCESS_KEY_ID ",
      { "Ref": "AutomationUserAK" }
    ]] }
  },
  "2-SetSK" : {
    "command" : { "Fn::Join" : ["", [
      "setx AWS_SECRET_ACCESS_KEY ",
      { "Ref": "AutomationUserSK" }
    ]] }
  },
  "3-Pullcode" : {
    "command" : "aws s3 sync s3://some-s3-bucket d:/dev/ --debug"
  }
}

The first 3 commands (Tester, SetAK, SetSK) work just fine. The last one (Pullcode) fails, every time.

So at this point I'm assuming I'm going at it wrong. Maybe I need to configure a specific IAM for the CF stack, maybe I should use setx ... /M, maybe a million other options, but since this trial and error has been going for the entire length of my work day and then some I figured it won't hurt to ask.

  • I saw a similar issue (albeit apparently on Linux) here: serverfault.com/questions/731109/… and possible a couple others on AWS's forums. Seems like this might actually be a bug, which would mean I have to do some kind of alternative, like putting the code in a ZIP file on a public S3 URL, downloading it with the files feature and then unzipping it with a shell command. This is not great. – motig88 Jul 15 '17 at 13:43
1

Each of your commands is run in a separate PowerShell, hence setx AWS_ACCESS_KEY_ID sets the variable in one shell instance and exits. Then setx AWS_SECRET_ACCESS_KEY sets another variable in a new shell and exits. Then you run aws s3 sync in yet another shell where none of the previous variables exist and it fails because the credentials variables do not exist in that shell.

Instead of calling it from Metadata you can do something like:

"UserData": {
  "Fn::Base64": { "Fn::Join": ["", [
    "<script>\n",
    "cfn-init.exe -v -s [...]\n",
    "setx AWS_ACCESS_KEY_ID ", { "Ref": "AutomationUserAK" }, "\n",
    "setx AWS_SECRET_ACCESS_KEY ", { "Ref": "AutomationUserSK" }, "\n",
    "aws s3 sync s3://some-s3-bucket d:/dev/\n",
    "</script>"
  ] ] }
}

In this case it will all be run in the same shell instance and the keys will be still set when you call aws s3 sync.

Use IAM Roles instead

However I have to say that passing access and secret keys to CloudFormation template is a bad bad practice. Instead the CloudFormation template should create IAM Role with the appropriate permissions to access the S3 bucket and assign it to the EC2 instance. The aws tool on the instance will than have transparent access to S3 without any need to supply access and secret keys. Something like this should do for your CloudFormation template:

First create the IAM Role with S3 access policy (adjust the policy for your needs):

"InstanceRole": {
  "Type": "AWS::IAM::Role",
  "Properties": {
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": { "Service": [ "ec2.amazonaws.com" ] },
          "Action": [ "sts:AssumeRole" ]
        }
      ]
    },
    "Path": "/",
    "Policies": [
      { 
        "PolicyName": "S3_Access",
        "PolicyDocument": {
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [ "s3:List*", "s3:Get*" ],
              "Resource": "*"
            }
          ]
        }
      }
    ]
  }
},

Then create an IAM Instance Profile that refers to the above IAM Role...

"InstanceProfile": {
  "Type": "AWS::IAM::InstanceProfile",
  "Properties": {
    "Path": "/",
    "Roles": [ { "Ref": "InstanceRole" }
    ]
  }
},

And finally create the EC2 Instance that gets that IAM Profile assigned (and in turn IAM Role with the S3 access policy).

"Instance": {
  "Type": "AWS::EC2::Instance",
  "Properties": {
    "IamInstanceProfile": { "Ref": "InstanceProfile" },
    [...],
    "UserData": {
      "Fn::Base64": { "Fn::Join": ["", [
        "<script>\n",
        "cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" }, " -r Instance --region ", { "Ref" : "AWS::Region" }, "\n",
        "</script>"
      ] ] }
    }
  },
  "Metadata": {
    "AWS::CloudFormation::Init": {
      "config": {
        "commands" : {
          "Pullcode" : {
            "command" : "aws s3 sync s3://some-s3-bucket d:/dev/ --debug"
          }
        }
      } 
    }
  }
},

When using IAM Roles the AWS credentials are dynamically generated and transparently obtained by the aws command. This is much more secure and a preferable way to give EC2 instances access to AWS resources.

Learn more about EC2 Instance Roles for example here: https://aws.nz/best-practice/ec2-instance-roles/

Hope that helps :)

  • Thanks @MLu - saved again. I had a feeling that passing the keys via CFN templates is a bad idea but I was running out of ideas. This would be the next step to try, but you've cleared up the right way of doing it and it worked. Thanks again! – motig88 Jul 17 '17 at 9:05
  • @motig88 - I'm always happy to help :) BTW if you're still working on your 3-server setup that we discussed in the other question be aware that you can share the InstanceRole and InstanceProfile between them. You don't need to create separate ones for each instance. – MLu Jul 17 '17 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.