Unable to create Kubernetes resources with terraform

Question

I have a working GKE Kubernetes cluster. I'm trying to use terraform to deploy new resources to it, but Terraform apply returns the following:

Error: Error applying plan:

2 error(s) occurred:

* kubernetes_pod.test: 1 error(s) occurred:

* kubernetes_pod.test: pods is forbidden: User "client" cannot create pods in the namespace "default"
* module.helm.kubernetes_service_account.tiller: 1 error(s) occurred:

* kubernetes_service_account.tiller: serviceaccounts is forbidden: User "client" cannot create serviceaccounts in the namespace "kube-system"

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

My terraform files are as follows:

GKE Resources main.tf

data "terraform_remote_state" "project" {
  backend = "gcs"

  config {
    bucket = "<bucket>"
    prefix = "terraform/testing-project"
  }
}

data "terraform_remote_state" "gke" {
  backend = "gcs"

  config {
    bucket = "<bucket>"
    prefix = "terraform/testing-gke-cluster"
  }
}

locals {
  host                   = "${data.terraform_remote_state.gke.endpoint}"
  client_certificate     = "${base64decode(data.terraform_remote_state.gke.client_certificate)}"
  client_key             = "${base64decode(data.terraform_remote_state.gke.client_key)}"
  cluster_ca_certificate = "${base64decode(data.terraform_remote_state.gke.cluster_ca_certificate)}"
}

provider "kubernetes" {
  host                   = "${local.host}"
  client_certificate     = "${local.client_certificate}"
  client_key             = "${local.client_key}"
  cluster_ca_certificate = "${local.cluster_ca_certificate}"
}

module "helm" {
  source                 = "../../../Modules/GKE/Helm/"
  host                   = "${local.host}"
  client_certificate     = "${local.client_certificate}"
  client_key             = "${local.client_key}"
  cluster_ca_certificate = "${local.cluster_ca_certificate}"
}


resource "kubernetes_pod" "test" {
  metadata {
    name = "terraform-example"
  }

  spec {
    container {
      image = "nginx:1.7.9"
      name  = "example"

      env {
        name  = "environment"
        value = "test"
      }
    }
  }
}

And the helm -module:

resource "kubernetes_service_account" "tiller" {
  automount_service_account_token = true

  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller"
  }

  role_ref {    
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }

  subject {
    kind      = "ServiceAccount"
    name      = "${kubernetes_service_account.tiller.metadata.0.name}"
    api_group = ""
    namespace = "${kubernetes_service_account.tiller.metadata.0.namespace}"
  }
}

# initialize Helm provider
provider "helm" {
  install_tiller = true
  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  tiller_image = "gcr.io/kubernetes-helm/tiller:v2.11.0"

  kubernetes {
  host                   = "${var.host}"
  client_certificate     = "${var.client_certificate}"
  client_key             = "${var.client_key}"
  cluster_ca_certificate = "${var.cluster_ca_certificate}"
  } 
}

I've been trying to search for a solution and my best guess is that is has something to do with me previously using my google account to manage resources with kubectl and kubernetes having problems with the terraform service account now but I don't know how to solve it. The terraform service account has owner and editor roles in the project.

Answer 1

You need to assign a Role Binding or a Cluster Role Binding to the user "client" for the above to work using RBAC. When you enable legacy authorization, you are turning ABAC on which is less secure

Answer 2

Apparently this started to work when I enabled Legacy Authorisation. If someone could explain why, I could mark that as an answer.