Whenever I try to run kubectl logs on azure, I get the following error:

Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log )

A similar error is returned for kubectl exec, kubectl attach and most helm commands:

error: unable to upgrade connection: Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy)

This is a default azure aks installation. Problems started after upgrading from 1.8.10 to 1.9.x

kubectl api-versions

admissionregistration.k8s.io/v1alpha1 admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1beta1 apps/v1 apps/v1beta1 apps/v1beta2 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 batch/v1 batch/v1beta1 certificates.k8s.io/v1beta1 events.k8s.io/v1beta1 extensions/v1beta1 networking.k8s.io/v1 policy/v1beta1 rbac.authorization.k8s.io/v1 rbac.authorization.k8s.io/v1beta1 storage.k8s.io/v1 storage.k8s.io/v1beta1 v1

kubectl version

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean",BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean",BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

  • As a temporary workaround for development i logged into the node and changed the /etc/defaults/kubelet file to use --authorization-mode=AllowAll instead of WebHook. I can kubectl logs now, but exec, attach and helm still don't work and i don't think this is a viable solution to the problem. – Markus Dresch Jun 7 '18 at 13:21

i managed to create a workaround by applying this yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
  name: anonymous-logs-role
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["create", "get"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
  name: anonymous-logs-binding
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: anonymous-logs-role
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:anonymous

this way logs, exec and attach work and the api server doesn't expose any information. not sure if there are any drawbacks...


This command will give your user (you) a cluster-admin role.

kubectl create clusterrolebinding me-cluster-admin \
    --clusterrole=cluster-admin \
  • That's right, but this gives cluster admin access to everyone - even on the web API. The yaml i posted doesn't give anonymous access to the web API. Still not sure if my solution is safe to use in production though. – Markus Dresch Jun 11 '18 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.