GCP Service Account can't access IAM operations with permissions

Question

I'm using Terraform to automate a lot of my GCP management because clicking is bad. I've got a "shared services" project that I'm trying to use to manage other projects. I'm trying to setup a new environment in another project and need a service account in the shared services project to manage the resources there. (I don't want to by-hand create a new service account for each project)

I'm trying to create a service account in the new project using the shared services service account. The shared services account has organization-level permissions, but I've been trying to add project-level permissions to fix the issue.

Here's the output of gcloud projects get-iam-policy newproject (irrelevant info removed, renamed):

bindings:
- members:
  - serviceAccount:terraform@shared-services-####.iam.gserviceaccount.com
  role: roles/editor
- members:
  - serviceAccount:terraform@shared-services-####.iam.gserviceaccount.com
  role: roles/iam.serviceAccountAdmin
- members:
  - serviceAccount:terraform@shared-services-####.iam.gserviceaccount.com
  role: roles/iam.serviceAccountKeyAdmin
- members:
  - serviceAccount:terraform@shared-services-####.iam.gserviceaccount.com
  role: roles/resourcemanager.projectIamAdmin
etag: BwWGI28ti_w=
version: 1

Here's the output I get attempting to run a test command:

> gcloud iam service-accounts get-iam-policy [email protected] --project=newproject --impersonate-service-account=terraform@shared-services-####.iam.gserviceaccount.com
WARNING: This command is using service account impersonation. All API calls will be executed as [terraform@shared-services-####.iam.gserviceaccount.com].
ERROR: (gcloud.iam.service-accounts.get-iam-policy) PERMISSION_DENIED: The caller does not have permission

The permissions reference states that roles/iam.serviceAccountAdmin provides this permission.

Here's the output that Terraform gives me (I know it's a different operation):

Error 403: Permission iam.serviceAccounts.get is required to perform this operation on service account projects/-/serviceAccounts/[email protected]., forbidden

I did create the new service account by hand for this specific case because I haven't setup the rest of the infrastructure yet (which would create the account as part of its process). I wanted to make sure this worked.

Any ideas? As far as I can tell, I've granted the permissions it's telling me I need. Help?

Edit: Oh, I checked out trying the API, and I get a 403 as my user account, which should have organization admin:

{
  "error": {
    "code": 403,
    "message": "Permission iam.serviceAccounts.get is required to perform this operation on service account projects/-/serviceAccounts/[email protected].",
    "status": "PERMISSION_DENIED"
  }
}

I am thoroughly confused

Answer 1

You probably used a google_project_iam_policy resource incorrectly, and overwrote the default IAM policy configuration for the project with an incorrect policy (don't ask how I know this...)

google_project_iam_policy is a very dangerous resource in Terraform, and the docs do not sufficiently emphasize how dangerous it is. The problem is that setting the IAM Policy replaces your project's entire IAM configuration with the IAM policy you define. All the default, auto-created service account permissions get wiped out unless you specifically included them in your policy definition.

It is possible to fix your project, but not easy. You need to find all the service accounts that your project needs, and add the correct permissions. Error output from TF_LOG=TRACE terraform apply can guide you. The default project IAM policy should look something like the policy below, though it will differ based on which APIs you have enabled and which Google Cloud features are in use. (policy sanitized with xxxxx replacing project ID)

bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:[email protected]
  - serviceAccount:[email protected]
  - serviceAccount:terraform-service-account@your-terraform-project.iam.gserviceaccount.com
  role: roles/editor
- members:
  - user:[email protected]
  role: roles/owner
- members:
  - serviceAccount:terraform-service-account@your-terraform-project.iam.gserviceaccount.com
  role: roles/servicenetworking.networksAdmin
- members:
  - serviceAccount:[email protected]
  role: roles/servicenetworking.serviceAgent
etag: BwWc0THMaHA=
version: 1

If you are getting this error, run gcloud projects get-iam-policy your-project-name and see what's missing. In all likelihood, the policy change wiped out your owner role, and roles for the default service accounts (the ones that include your project ID in the name).

Summary: if you're using Terraform to manage IAM in Google Cloud Platform, you should generally NOT be using resource google_project_iam_policy, unless you are an expert at hand-writing Google IAM policies. If you must use it, before you begin, run gcloud projects get-iam-policy your-project-name and save the results so you can see what your IAM policy looked like before you broke it.

Answer 2

Agree with previous answer, just noting that you can view all of the roles that were deleted in IAM -> View Resources. Once there, check the project that you accidentally nuked, click Activity, and each change until you find your super-destructive one. Should be much easier to go through there and add the changes back.