I am trying to set up our private Kubernetes cluster in Google Cloud to connect to Google Container Engine. I'm able to deploy sample images in the cluster without a problem, e.g. gcr.io/google-samples/hello-app:2.0. But when I try to deploy one of our own images, i.e. gcr.io/[OUR_PROJECT_ID]/test-image:1.0, then I get ImagePullBackOff errors showing up in Kubernetes.
ImagePullBackOff never shows any details about what caused the error. I tried logging in to one of the cluster's nodes directly (as recommended in the troubleshooting section of Google's docs), but I can't download the image from there either, even though pulling the image works fine from a public cluster. The node just doesn't seem to be a realistic troubleshooting environment, since although I know the demo image works, even that fails from inside the node:
$ docker pull gcr.io/google-samples/hello-app:2.0 Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/create?fromImage=gcr.io%2Fgoogle-samples%2Fhello-app&tag=2.0: dial unix /var/run/docker.sock: connect: permission denied
It also works fine locally. So something about the private cluster is blocking it.
How can I get more details on why the image pull failed? It would be great to see the error that Docker is actually returning.
And of course if anyone knows the likely problem here and how to resolve it, I'm all ears. Private Google Access is already enabled on the network, and the cluster's service account already has access to the storage bucket used by GCE, so I don't think those are the issues.