0

In order to dynamically build a configuration file, I need to query EC2 from an instance to retrieve information on existing instances. I use the aws CLI with the ec2 describe-instances flag. The instance I'm running the command from is attached to a role that permits such queries:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInstances",
                "ec2:DescribeRegions",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeTags"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

This works perfectly fine when the instance is started, but when I try to fetch data from a user-data file, I get:

A client error (AuthFailure) occurred when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials

I read that this could be caused by a not-time-synchronized instance, so I added openntpd to the user-data file, before calling awscli, but still get the same error. Within the userdata, I successfully access to s3 and route53 using awscli:

aws s3 cp s3://s3test/foobar.yml playbook.yml

And just to be sure the IAM policy was ok, I also tried with AWS's EC2ReadOnly policy, and got the same result.

The query is done this way:

region="eu-central-1"

for ip in $(aws ec2 describe-instances --debug --filters 'Name=tag:Name,Values=rabbitmq' --region "$region"|jq -r '.Reservations[].Instances[].NetworkInterfaces[].PrivateIpAddresses[].PrivateIpAddress')
do
    # [stuff to be done]
done

Please note this is not an authentication failure, I am using roles to allow the instance to query EC2.

Anyone around using an awscli EC2 access and willing to share experience?

  • Have you tried putting a delay in your user-data script before the aws command? – EEAA Aug 1 '16 at 12:13
  • I've added a sleep 10 after the ntp synchronization, should be enough but it might not hurt to try increasing it... – iMil Aug 1 '16 at 12:16
  • It just seems like the command may be running before credentials are deployed to the metadata service. – EEAA Aug 1 '16 at 12:17
  • 1
    Yah, interesting. Honestly I'm out of ideas. I've not seen this before and I use instance roles and user-data aws commands extensively. Hope you're able to get to the bottom of it! – EEAA Aug 1 '16 at 12:46
  • 1
    Good catch! Go ahead and post that as an answer to your own question, and then accept it. – EEAA Aug 1 '16 at 15:23
1

As I said in the comment section, I finally figured out this issue was related to a missing $PATH. In my userdata script, awscli is installed via pip, which installs it in /usr/local/bin. The aws command itself is actually found as it is executed, but there must be a sub-process called within awscli when the describe-instances function is called that does not know about /usr/local/bin, which leads to the previously mentionned error. The trivial fix for this is to prepend $PATH with /usr/local/bin in the userdata script:

PATH=/usr/local/bin:$PATH; export PATH

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.