1

Goal: Re-initialize a Terraform project to another workstation.

Problem: Terraform wants to recreate one my TLS keys because it thinks the private key has a diff. However since I cannot see the diff since it's a sensitive file, I'm lost.


  • My companies' Terraform repo installs TLS/SSL certs/keys into AWS Load Balancers.
  • I had to restore a Terraform project to a new workstation (old workstation is retired).
  • The cert and chains are stored in the repo.
  • The private keys themselves are NOT checked into the git repo, but we keep multiple secure backups.

Certs are being declared like this:

resource "aws_iam_server_certificate" "test1_cert_renewed" {
  name              = "test1_cert_renewed"
  certificate_body  = file("${path.cwd}${var.test1_cert_renewed_cert}")
  private_key       = file("${path.cwd}${var.test1_cert_renewed_key}")
  certificate_chain = file("${path.cwd}${var.test1_cert_renewed_chain}")

  lifecycle {
    create_before_destroy = true
  }
}

Everything is going smoothly to restore the Terraform project to a working order on the new workstation. However it wants to recreate my cert installs because it thinks the private key is different.

On a plan, it just says it'll recreate due to:

~ private_key = (sensitive value)

This is only affecting ONE of the 5 different cert bundles so it's something specific about this file.

  1. Is it possible to see more details in the Terraform output, so I can at least see part of the diff snippet?

  2. Any advice on how I can diff what is in state vs what I have in the file?

I've tried terraform state pull and I do see output, but it's an encoded string. Not entirely sure how to decode (if possible).

    {
      "mode": "managed",
      "type": "aws_iam_server_certificate",
      "name": "<<SCRUBBED>>",
      "provider": "provider.aws",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "arn": "arn:aws:iam::<<SCRUBBED>>",
            "certificate_body": "<<SCRUBBED>>",
            "certificate_chain": "<<SCRUBBED>>",
            "id": "ASCA4DNFUOPYAQICRDU7O",
            "name": "<<SCRUBBED>>",
            "name_prefix": null,
            "path": "/",
            "private_key": "<<SCRUBBED string that was here (looked like encoded string), but it wasn't private key text>>"
          },
          "private": "<<SCRUBBED 5 character string SCRUBBED>>"
        }
      ]
    }

More details:

I've already checked obvious things like newline at end/start of file:

-----END PRIVATE KEY-----

versus

-----END PRIVATE KEY-----

1

If there is a diff in TF, it's because a field in the state which is considered to be important is out of sync with the infrastructure a resource renders.

There may be something different about this resource in AWS - perhaps it was changed outside of TF? That would explain why this one certificate resource is having issues.

All of that aside, take a backup of your TF state, then delete this resource from the state using:

terraform state rm <resource path>

After deletion, import the resource you want, using resource path and the AWS object ID (in the state file you posted, this would be ID ASCA4DNFUOPYAQICRDU7O). This will update TF with the state as it is described in AWS, effectively pushing changes in the opposite direction. If you would like a hint on what the import command should look like, please provide the manifest code that's relevant to this resource, or display the current path of an already created resource via:

terraform state list | grep test1_cert_renewed

Generally, you need the path and ID to import. The following may be enough of a hint if you're not using any submodule structure within this manifest:

terraform import aws_iam_server_cerificate.test1_cert_renewed ASCA4DNFUOPYAQICRDU7O
| improve this answer | |
  • Thanks, yes it seems the local file is different versus the state/actual resource. We don't save the private keys in the repo so it was restored from backup. 4 of the 5 worked fine immediately but this one it thinks there is a diff. Regarding your request to re-create the state from the applied resource: this won't work because the next time I apply it'll still think there is a diff with my private key file. Goal here is to edit my local key file to match whatever it expects but that's hidden -- if that's not possible, I'll just delete/recreate the resource during the next downtime window. – Rino Bino Oct 8 at 19:34
  • If you can't know what the existing key is, then recreating the resource seems like the only way for you to resolve the diff. – Spooler Oct 8 at 19:41
  • Appreciate the input. I'll recreate it – Rino Bino Oct 8 at 22:46
1

As you've seen, the normal terraform plan output hides the actual value for any attribute that is marked as being "sensitive" in the schema for a resource type.

However, that hiding of the value is purely cosmetic in order to avoid the value being saved in logs of the Terraform run, etc. You can view it if you save the plan to a plan file and then ask Terraform to render it as JSON:

terraform plan -out=tfplan
terraform show -json tfplan

The JSON plan output doesn't have a human-readable layout by default, so you may wish to pipe it through a tool like jq to interpret into a readable shape and possibly -- if you are familiar with the jq expression syntax -- extract only the specific part you care about.

Note here that the tfplan file created by the first command is a binary file in a format significant only to Terraform, but it does contain your secret key value and so you should be careful to run this command only on a system where that file being on disk will not lead to the value being compromised. Similarly, the JSON plan output will contain the sensitive values unobscured, so you should make sure nobody is "looking over your shoulder" at your screen if you intend to view it in your terminal.

| improve this answer | |
  • Thank you very much. This seems to show the same data in the output that terraform state pull shows. Unfortunately the field I need is displaying just as a short encoded string "private_key":"d51c97fyzv934c85a6x2c959a9040zb2e471650c" (yes, that's scrubbed) rather than the actual PEM key that I'm trying to diff against. I think I may need to just recreate the resource during the next downtime window because I cannot seem to figure out the diff on this file. – Rino Bino Oct 9 at 21:02
  • I think this behavior you're seeing is a detail of this particular resource type implementation in the AWS provider. I'm not familiar with the details of this resource type, but it seems like the provider is transforming your PEM input into a hash of some sort. If that's true then unfortunately you won't be able to view it because the original value was discarded by the provider before returning the checksum in its place in the plan. – Martin Atkins Oct 9 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.