Private s3 access from ec2 user-data script

Question

I'm attempting to write an ec2 user-data script that will pull a file down from a private s3 bucket. The ec2 instances are located in multiple regions, which I believe eliminates the possibility of using a bucket policy to restrict access to a VPC (which was working well in my prototype, but broke in the second region).

Based on advice here and elsewhere, the approach that seems like it should work is giving the ec2 instance an IAM role with access to that s3 bucket. And in fact, this almost seems to work for me. However, just not at the time the user-data script is running.

My user-data script has a while loop that checks for the existence of the file I'm trying to download from s3, and will keep retrying for 5 minutes. If I log in manually during that window and run the exact aws command manually in that 5 minute window, the user-data script succeeds as expected, but it never succeeds on its own.

apt install -y python-minimal
apt install -y unzip
wget "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip"
unzip awscli-bundle.zip
./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

mkdir -p /opt/myapp
cd /opt/myapp

n=0
while [ ! -f app.tar.gz ]; do
    aws s3 cp --region us-west-1 "s3://bucket_name/app.tar.gz" app.tar.gz
    n=$[$n+1]
    [ $n -ge 60 ] && break
    sleep 5
done

tar -zxf app.tar.gz
./bin/startapp

That is a distilled version of my user-data script. If I'm able to login and run that same aws command manually, I believe the IAM role must be correct; but I don't understand what else might be going wrong. When the aws command is run from the user-data script, the error is: fatal error: 'An error occurred (403) when calling the HeadObject operation: Forbidden'

Answer 1

I eventually solved the issue, but the problem was impossible to see from what I posted. I distilled the user-data script into something I could post here, but it is actually much longer. The key portion not posted is that the app ultimately started by this process needs access to Amazon's SES. When I posted this question, those environment variables were at the top of the script, and the S3 call was using them and not the assigned role. Once I moved the environment variables to just before the app starts (importantly, after that S3 download command), the user-data script worked as expected.

The important part worth capturing is that although the IAM role is assigned to the instance, and the AWS cli will use those credentials automatically, the credentials are used just like user credentials for things like calls to S3. Meaning setting other credentials will absolutely override those from the role. Before digging into this issue I was under the impression the S3 service might have a way to validate the caller and the caller's role when the request is internal to AWS.