aws ec2 describe-instances not allowed in user-data?

Question

In order to dynamically build a configuration file, I need to query EC2 from an instance to retrieve information on existing instances. I use the aws CLI with the ec2 describe-instances flag. The instance I'm running the command from is attached to a role that permits such queries:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInstances",
                "ec2:DescribeRegions",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeTags"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

This works perfectly fine when the instance is started, but when I try to fetch data from a user-data file, I get:

A client error (AuthFailure) occurred when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials

I read that this could be caused by a not-time-synchronized instance, so I added openntpd to the user-data file, before calling awscli, but still get the same error. Within the userdata, I successfully access to s3 and route53 using awscli:

aws s3 cp s3://s3test/foobar.yml playbook.yml

And just to be sure the IAM policy was ok, I also tried with AWS's EC2ReadOnly policy, and got the same result.

The query is done this way:

region="eu-central-1"

for ip in $(aws ec2 describe-instances --debug --filters 'Name=tag:Name,Values=rabbitmq' --region "$region"|jq -r '.Reservations[].Instances[].NetworkInterfaces[].PrivateIpAddresses[].PrivateIpAddress')
do
    # [stuff to be done]
done

Please note this is not an authentication failure, I am using roles to allow the instance to query EC2.

Anyone around using an awscli EC2 access and willing to share experience?

Answer 1

As I said in the comment section, I finally figured out this issue was related to a missing $PATH. In my userdata script, awscli is installed via pip, which installs it in /usr/local/bin. The aws command itself is actually found as it is executed, but there must be a sub-process called within awscli when the describe-instances function is called that does not know about /usr/local/bin, which leads to the previously mentionned error. The trivial fix for this is to prepend $PATH with /usr/local/bin in the userdata script:

PATH=/usr/local/bin:$PATH; export PATH