3

How can I set up AWS Glue using Terraform (specifically I want it to be able to spider my S3 buckets and look at table structures). A quick Google search came up dry for that particular service. The S3 bucket I want to interact with is already and I don't want to give Glue full access to all of my buckets.

I've submitted my solution Q & A style, but I'm interested to see if there are any thoughts on how I could have done it better.

5
  • What you tried?
    – peterh
    Aug 23, 2017 at 0:13
  • @peterh I submitted an answer with my best stab. Aug 23, 2017 at 3:20
  • 1
    @russellpierce I believe Peterh's question came as a result of this question being shown in the review queue. Your answer is good and concise, but the question isn't - it's basically just a vehicle for the answer. If you could take the time to edit the question a bit, describing in more detail what you wanted to do and what your requirements are, that would be very helpful to other people who might find the question.
    – Jenny D
    Aug 28, 2017 at 9:59
  • Accepting your answer makes it look like you've solved your problem. Which means nobody else is likely to bother to formulate an answer. Also, as Jenny D said, your question needs to stand on its own, without reading the other answers, so I think the way you've structured this is unproductive.
    – womble
    Sep 7, 2017 at 0:52
  • @womble and Jenny D I've deselected my answer and made some minor revisions to the question itself in the hopes others will find this useful. I'd be glad to take edits on the question body as well. Sep 11, 2017 at 19:38

1 Answer 1

5

If you have recommendations on how to do this better, then please submit an answer so I can do better next time.

My example here will closely reflect the situation I was in. In particular, the S3 bucket I wanted to interact with was already defined and I didn’t want to give Glue full access to all of my buckets.

The first component is the role itself. Amazon recommends the particular name I use in this section so that the role can be passed from console users to the service. Check out the IAM Role Section of the Glue Manual in the References section if that isn’t acceptable. The other thing that was different from a boilerplate “Assume Role“ was “Principal” and “Service”.

resource "aws_iam_role" "glue" {
  name = "AWSGlueServiceRoleDefault"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

The next component was to attach the AWSGlueServiceRole managed policy to the role. Amazon pre-defines this so that the role has almost all of the permissions it needs in order to work out of the gate.

resource "aws_iam_role_policy_attachment" "glue_service" {
    role = "${aws_iam_role.glue.id}"
    policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

If you don’t have a policy already defined for your S3 bucket, then you can define your policy and attach it to this glue role all in the same block, like this:

resource "aws_iam_role_policy" "my_s3_policy" {
  name = "my_s3_policy"
  role = "${aws_iam_role.glue.id}"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::my_bucket",
        "arn:aws:s3:::my_bucket/*"
      ]
    }
  ]
}
EOF
}

If, like me, you defined that policy but had already attached it to another role, then you can re-use it and attach it to the glue role as well like this:

resource "aws_iam_role_policy" "glue_service_s3" {
 name = "glue_service_s3"
    role = "${aws_iam_role.glue.id}"
    policy = "${aws_iam_role_policy.my_s3_policy.policy}"
}

The text you’d change here to match your configuration would be ‘my_s3_policy’ for the policy option/key.

My answer here replicated in part in my Medium post.

1
  • 1
    Terraform also has aws_glue_catalog_database and aws_glue_catalog_table resources these days
    – Davos
    Jul 31, 2019 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.