I'm attempting to write an ec2 user-data script that will pull a file down from a private s3 bucket. The ec2 instances are located in multiple regions, which I believe eliminates the possibility of using a bucket policy to restrict access to a VPC (which was working well in my prototype, but broke in the second region).

Based on advice here and elsewhere, the approach that seems like it should work is giving the ec2 instance an IAM role with access to that s3 bucket. And in fact, this almost seems to work for me. However, just not at the time the user-data script is running.

My user-data script has a while loop that checks for the existence of the file I'm trying to download from s3, and will keep retrying for 5 minutes. If I log in manually during that window and run the exact aws command manually in that 5 minute window, the user-data script succeeds as expected, but it never succeeds on its own.

apt install -y python-minimal
apt install -y unzip
wget "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip"
unzip awscli-bundle.zip
./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

mkdir -p /opt/myapp
cd /opt/myapp

while [ ! -f app.tar.gz ]; do
    aws s3 cp --region us-west-1 "s3://bucket_name/app.tar.gz" app.tar.gz
    [ $n -ge 60 ] && break
    sleep 5

tar -zxf app.tar.gz

That is a distilled version of my user-data script. If I'm able to login and run that same aws command manually, I believe the IAM role must be correct; but I don't understand what else might be going wrong. When the aws command is run from the user-data script, the error is: fatal error: 'An error occurred (403) when calling the HeadObject operation: Forbidden'

  • You could use S3 Cross Region Replication so your data is in multiple regions, but that increases your storage cost. I looked at EC2 life cycle to try to work out when IAM role is attached but can't find anything. I wonder if you can do some of test in your user data script to check that? Query instance data perhaps? That would give more information to help solve this.
    – Tim
    Sep 28 '18 at 18:45
  • 1
    @Tim Both /latest/meta-data/iam/info and /latest/meta-data/iam/security-credentials/<name> when run from the user-data script return what I would expect (run just before the while loop)...unless I'm misunderstanding your question.
    – Chris
    Sep 28 '18 at 20:31
  • That's what I meant. So your instance has the correct role when the script is running, but can't contact S3 at all? Perhaps you can test connectivity in the script? I'd also consider an S3 gateway in your VPC, with the route table set up to route to it.
    – Tim
    Sep 29 '18 at 0:20

I eventually solved the issue, but the problem was impossible to see from what I posted. I distilled the user-data script into something I could post here, but it is actually much longer. The key portion not posted is that the app ultimately started by this process needs access to Amazon's SES. When I posted this question, those environment variables were at the top of the script, and the S3 call was using them and not the assigned role. Once I moved the environment variables to just before the app starts (importantly, after that S3 download command), the user-data script worked as expected.

The important part worth capturing is that although the IAM role is assigned to the instance, and the AWS cli will use those credentials automatically, the credentials are used just like user credentials for things like calls to S3. Meaning setting other credentials will absolutely override those from the role. Before digging into this issue I was under the impression the S3 service might have a way to validate the caller and the caller's role when the request is internal to AWS.

  • 1
    So, to summarize your post: while your EC2 instance had a role with sufficient permissions, you had overridden those permissions in the script.
    – Tim
    Sep 30 '18 at 0:53
  • This is a good answer to a real world problem. Why are you using access keys for SES instead of an IAM role? Never store credentials on your instance in any form. Oct 3 '18 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.