3

How can I set up AWS Glue using Terraform (specifically I want it to be able to spider my S3 buckets and look at table structures). A quick Google search came up dry for that particular service. The S3 bucket I want to interact with is already and I don't want to give Glue full access to all of my buckets.

I've submitted my solution Q & A style, but I'm interested to see if there are any thoughts on how I could have done it better.

| improve this question | | | | |
  • What you tried? – peterh - Reinstate Monica Aug 23 '17 at 0:13
  • @peterh I submitted an answer with my best stab. – russellpierce Aug 23 '17 at 3:20
  • 1
    @russellpierce I believe Peterh's question came as a result of this question being shown in the review queue. Your answer is good and concise, but the question isn't - it's basically just a vehicle for the answer. If you could take the time to edit the question a bit, describing in more detail what you wanted to do and what your requirements are, that would be very helpful to other people who might find the question. – Jenny D Aug 28 '17 at 9:59
  • Accepting your answer makes it look like you've solved your problem. Which means nobody else is likely to bother to formulate an answer. Also, as Jenny D said, your question needs to stand on its own, without reading the other answers, so I think the way you've structured this is unproductive. – womble Sep 7 '17 at 0:52
  • @womble and Jenny D I've deselected my answer and made some minor revisions to the question itself in the hopes others will find this useful. I'd be glad to take edits on the question body as well. – russellpierce Sep 11 '17 at 19:38
5

If you have recommendations on how to do this better, then please submit an answer so I can do better next time.

My example here will closely reflect the situation I was in. In particular, the S3 bucket I wanted to interact with was already defined and I didn’t want to give Glue full access to all of my buckets.

The first component is the role itself. Amazon recommends the particular name I use in this section so that the role can be passed from console users to the service. Check out the IAM Role Section of the Glue Manual in the References section if that isn’t acceptable. The other thing that was different from a boilerplate “Assume Role“ was “Principal” and “Service”.

resource "aws_iam_role" "glue" {
  name = "AWSGlueServiceRoleDefault"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

The next component was to attach the AWSGlueServiceRole managed policy to the role. Amazon pre-defines this so that the role has almost all of the permissions it needs in order to work out of the gate.

resource "aws_iam_role_policy_attachment" "glue_service" {
    role = "${aws_iam_role.glue.id}"
    policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

If you don’t have a policy already defined for your S3 bucket, then you can define your policy and attach it to this glue role all in the same block, like this:

resource "aws_iam_role_policy" "my_s3_policy" {
  name = "my_s3_policy"
  role = "${aws_iam_role.glue.id}"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::my_bucket",
        "arn:aws:s3:::my_bucket/*"
      ]
    }
  ]
}
EOF
}

If, like me, you defined that policy but had already attached it to another role, then you can re-use it and attach it to the glue role as well like this:

resource "aws_iam_role_policy" "glue_service_s3" {
 name = "glue_service_s3"
    role = "${aws_iam_role.glue.id}"
    policy = "${aws_iam_role_policy.my_s3_policy.policy}"
}

The text you’d change here to match your configuration would be ‘my_s3_policy’ for the policy option/key.

My answer here replicated in part in my Medium post.

| improve this answer | | | | |
  • 1
    Terraform also has aws_glue_catalog_database and aws_glue_catalog_table resources these days – Davos Jul 31 '19 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.