Stackdriver missing GKE Logs

Question

I brought up a new Vault cluster using Terraform (https://github.com/sethvargo/vault-on-gke) and everything is working fine with Vault. However I can't seem to see any Container, Node or Pod logs in Stackdriver. The other GKE Cluster we have that was brought up manually is working fine.

I can view the logs on the pods using kubectl logs without a problem.

Existing cluster that works fine

Terraformed vault cluster

I'm completely stumped as to why it's not working. Stackdriver is enabled for the cluster, and the beta Stackdriver box is ticked - same as the working cluster. The Compute service account has Editor role, same in both account.

Any thoughts?

Answer 1

You need to make sure that your kubernetes cluster has the correct permissions for logging and monitoring.

To have a properly running cluster, the kubernetes master needs the following oauth scopes (the latter two are for stackdriver):

• compute-rw (https://www.googleapis.com/auth/compute)
• storage-ro (https://www.googleapis.com/auth/devstorage.read_only)
• logging-write (https://www.googleapis.com/auth/logging.write)
• monitoring (https://www.googleapis.com/auth/monitoring)

In addition, the service account of the nodes in the kubernetes cluster need the following permissions:

• roles/logging.logWriter (https://cloud.google.com/iam/docs/understanding-roles#stackdriver_logging_roles)
• roles/monitoring.metricWriter (https://cloud.google.com/iam/docs/understanding-roles#stackdriver_monitoring_roles)

For more information see:

• https://www.terraform.io/docs/providers/google/r/container_cluster.html#oauth_scopes
• https://www.terraform.io/docs/providers/google/r/container_cluster.html#service_account

Answer 2

If you are looking for the equivalent of kubectl logs, you should be looking in 'kubernetes pod' or 'kubernetes container' for the logs. 'kubernetes cluster' shows you cluster activity such as resource patches and updates.

The next thing to check is to make sure that logging shows 'Enabled v2(beta)' and that the stackdriver agent is properly running in your cluster using 'kubectl get po -n kube-system | grep fluentd'

If all the above checks out, I recommend creating a public issue tracker as there is likely something wrong with either the cluster version of the beta logging