How to add lifecycle rule to an existing AWS S3 bucket with Terraform

Question

I have an existing S3 bucket and I wish to add "folders" and lifecycle_rules to it.

(I say "folders" because that is how they are represented at the client end, as they are accessed via a Storage Gateway.)

I can create the folders, for example on to hold quarterly backups, like:

resource "aws_s3_bucket_object" "quarterly" {
    bucket  = "${var.bucket_id}"
    acl     = "private"
    key     = "quarterly"
    source  = "/dev/null"
}

But if I try and then add a lifecycle rule, as follows

resource "aws_s3_bucket" "quarterly" {
    bucket  = "${var.bucket_id}"
    acl     = "private"

    lifecycle_rule {
        id      = "quarterly_retention"
        prefix  = "quarterly/"
        enabled = true
        tags {
            "rule"  = "quarterly"
        }

        expiration {
            days = 92
        }
    }
}

I get an error when I do terraform apply.

* aws_s3_bucket.quarterly: Error creating S3 bucket: BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
    status code: 409, request id: 702396A7D2FA28BA, host id: IJDA+vszRBYl4zmvW56dSnC2va2qpQXlfgeEL7X1QQHHv8eEaYKvSUCL0ZIj/VsdvQ2hkBLGjAY=

I want to create the bucket first and then add folders and lifecycle rules afterwards, rather than embed the lifecycle rules on creation.

Am I missing something, have I got it wrong?

Thanks for your help!

Answer 1

Creating the bucket first and incrementally updating the configuration should work fine, the net result is that if you were to delete the bucket by other means Terraform would recreate it with all of the rules in place.

It looks like you've lost your terraform.tfstate file so Terraform doesn't know that it has already created your bucket, or you created the bucket outside of Terraform in the first place, so it's trying to create it and failing. Terraform needs to "own" the bucket to be able to update its configuration, i.e. the lifecycle rules.

You should be able to import the existing bucket into your state file with something like

terraform import aws_s3_bucket.quarterly <your bucket ID>

See the bottom of https://www.terraform.io/docs/providers/aws/r/s3_bucket.html

Running Terraform should then show it just updating the lifecycle rules.

Answer 2

As per @bodgit, the method is to embed the lifecycle rules within the "aws_s3_bucket" resource and re-running "terraform apply".

lifecycle_rule clauses can be added to (or removed from) the resource and they are applied to the bucket.

I was looking to separate the rules from the bucket creation, so they could be actioned distinctly, but this'll do.

So, I define the bucket with:

resource "aws_s3_bucket" "bucket" {
    bucket      = "${replace(var.tags["Name"],"/_/","-")}"
    region      = "${var.aws_region}"

    tags        = "${merge(var.tags, map("Name", "${replace(var.tags["Name"],"/_/","-")}"))}"

    lifecycle_rule {
        id      = "quarterly_retention"
        prefix  = "quarterly/"
        enabled = true

        expiration {
            days = 92
        }
    }

}

When I run "terraform apply" the bucket is created with 1 rule.

I then edit my .tf file and add a 2nd lifecycle_rule.

resource "aws_s3_bucket" "bucket" {
    bucket      = "${replace(var.tags["Name"],"/_/","-")}"
    region      = "${var.aws_region}"

    tags        = "${merge(var.tags, map("Name", "${replace(var.tags["Name"],"/_/","-")}"))}"

    lifecycle_rule {
        id      = "quarterly_retention"
        prefix  = "quarterly/"
        enabled = true

        expiration {
            days = 92
        }
    }

    lifecycle_rule {
        id      = "permanent_retention"
        enabled = true
        prefix  = "permanent/"

        transition {
            days            = 1
            storage_class   = "GLACIER"
        }
    }

}

When I execute "terraform apply" again, the bucket has the 2nd rule added.

If I then edit the rule again, remove the rules, and run an "apply" again, the rules are gone.

If I then edit, add a rule back, and run apply again, that rule is there.

Thanks for the help!

Answer 3

At present AWS Provider v3.27 does not support externalizing the bucket configurations for AWS managed S3, it does allow it for S3 Outposts using s3control_bucket_lifecycle_configuration

For now all I can do is

  lifecycle {
    ignore_changes = [lifecycle_rule]
  }

To ignore changes that were applied after the fact. I use this technique quite a bit anyway for example I don't want my EC2 instance to be recreated if the AMI image gets updated.

However, if I am going to be just doing a smaller subset of the rules for example just set the expiration per prefix, I can use dynamic blocks.

For example

variable "cache_expiration_rules" {
  description = "Expiration rules for S3 cache"
  default     = []
  type        = list(object({ prefix = string, days = number }))
}

resource "aws_s3_bucket" "cache" {
  bucket        = local.cache_bucket_name
  acl           = "private"
  force_destroy = false

  lifecycle_rule {
    id                                     = "Common lifecycle rule"
    abort_incomplete_multipart_upload_days = 1
    enabled                                = true
    transition {
      days          = 1
      storage_class = "INTELLIGENT_TIERING"
    }
  }

  dynamic "lifecycle_rule" {
    for_each = var.cache_expiration_rules
    content {
      id      = "${lifecycle_rule.value["prefix"]} expiration in ${lifecycle_rule.value["days"]} days"
      enabled = true
      prefix  = lifecycle_rule.value["prefix"]
      expiration {
        days = lifecycle_rule.value["days"]
      }
    }
  }

}