0

In order to dynamically build a configuration file, I need to query EC2 from an instance to retrieve information on existing instances. I use the aws CLI with the ec2 describe-instances flag. The instance I'm running the command from is attached to a role that permits such queries:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInstances",
                "ec2:DescribeRegions",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeTags"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

This works perfectly fine when the instance is started, but when I try to fetch data from a user-data file, I get:

A client error (AuthFailure) occurred when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials

I read that this could be caused by a not-time-synchronized instance, so I added openntpd to the user-data file, before calling awscli, but still get the same error. Within the userdata, I successfully access to s3 and route53 using awscli:

aws s3 cp s3://s3test/foobar.yml playbook.yml

And just to be sure the IAM policy was ok, I also tried with AWS's EC2ReadOnly policy, and got the same result.

The query is done this way:

region="eu-central-1"

for ip in $(aws ec2 describe-instances --debug --filters 'Name=tag:Name,Values=rabbitmq' --region "$region"|jq -r '.Reservations[].Instances[].NetworkInterfaces[].PrivateIpAddresses[].PrivateIpAddress')
do
    # [stuff to be done]
done

Please note this is not an authentication failure, I am using roles to allow the instance to query EC2.

Anyone around using an awscli EC2 access and willing to share experience?

8
  • Have you tried putting a delay in your user-data script before the aws command?
    – EEAA
    Aug 1, 2016 at 12:13
  • I've added a sleep 10 after the ntp synchronization, should be enough but it might not hurt to try increasing it...
    – iMil
    Aug 1, 2016 at 12:16
  • It just seems like the command may be running before credentials are deployed to the metadata service.
    – EEAA
    Aug 1, 2016 at 12:17
  • 1
    Yah, interesting. Honestly I'm out of ideas. I've not seen this before and I use instance roles and user-data aws commands extensively. Hope you're able to get to the bottom of it!
    – EEAA
    Aug 1, 2016 at 12:46
  • 1
    Good catch! Go ahead and post that as an answer to your own question, and then accept it.
    – EEAA
    Aug 1, 2016 at 15:23

1 Answer 1

1

As I said in the comment section, I finally figured out this issue was related to a missing $PATH. In my userdata script, awscli is installed via pip, which installs it in /usr/local/bin. The aws command itself is actually found as it is executed, but there must be a sub-process called within awscli when the describe-instances function is called that does not know about /usr/local/bin, which leads to the previously mentionned error. The trivial fix for this is to prepend $PATH with /usr/local/bin in the userdata script:

PATH=/usr/local/bin:$PATH; export PATH

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .