AWS elastic beanstalk: Errno 404 downloading file from S3 on deployment

Question

I'm following the docs on fetching certificates from s3 when a new instance is deployed to elastic beanstalk. The instructions are fairly straightforward: create a config file under app-root/.ebextensions that reads:

Resources:
  AWSEBAutoScalingGroup:
    Metadata:
      AWS::CloudFormation::Authentication:
        S3Auth:
          type: "s3"
          buckets: ["elasticbeanstalk-us-west-2-123456789012"]
          roleName: 
            "Fn::GetOptionSetting": 
              Namespace: "aws:autoscaling:launchconfiguration"
              OptionName: "IamInstanceProfile"
              DefaultValue: "aws-elasticbeanstalk-ec2-role"
files:
  # Private key
  /etc/pki/tls/certs/server.key:
    mode: "000400"
    owner: root
    group: root
    authentication: "S3Auth"
    source: https://s3-us-west-2.amazonaws.com/elasticbeanstalk-us-west-2-123456789012/server.key

where the links point to the location on your certificate / key / whatever.

However, when I deploy my application bundle (zip), the build fails with [Errno 404] HTTP Error 404... suggesting that my files cannot be found at the specified location contrary to the fact that the files are at the precise specified location since I can aws s3 cp s3://<pasted-link-to-file> . them from my command line (same region of course).

Also, on the surface, it doesn't seem to be a permissions problem because I'd expect something like "Access Denied" if it was; the bucket policy grants the following actions to aws-elasticbeanstalk-ec2-role:

"Action": [
    "s3:ListBucket",
    "s3:ListBucketVersions",
    "s3:GetObject",
    "s3:GetObjectVersion"
]

I also attached full s3 access policy to aws-elasticbeanstalk-ec2-role in IAM.

But the fact that I can access the files from cli and aws-elasticbeanstalk-ec2-role cannot find them, suggests that a) something could be wrong with my setup: files on s3, s3 permissions, config in .ebextensions or b) aws docs on this matter are completely out of whack.

• This is a bit of a broad question but can anyone suggest what else could go wrong with this sort of setup?

Answer 1

Solved it by explicitly specifying:

roleName: "aws-elasticbeanstalk-ec2-role"

instead of:

roleName: 
  "Fn::GetOptionSetting": 
    Namespace: "aws:autoscaling:launchconfiguration"
    OptionName: "IamInstanceProfile"
    DefaultValue: "aws-elasticbeanstalk-ec2-role"

in .ebextensions/my_conf.config.

This could be because the config file key and the S3 policy refer to "roleName" and not "IamInstanceProfile".

Answer 2

After doing some troubleshooting, I realized the issue with :

"Fn::GetOptionSetting": 
  Namespace: "aws:autoscaling:launchconfiguration"
  OptionName: "IamInstanceProfile"
  DefaultValue: "aws-elasticbeanstalk-ec2-role"

This option returns the Instance Profile. However, to fetch credentials, we need the AWS Role associated with the instance profile.

Following command will return 404:

curl -vvv http://169.254.169.254/latest/meta-data/iam/security-credentials/{Instance-Profile}

However, following command works and returns temporary credentials:

curl -vvv http://169.254.169.254/latest/meta-data/iam/security-credentials/{Instance-ROLE}