We experienced a major outage with our production systems today. Without any user interaction most of the kubernetes pods went down with a "ImagePullBackOff" error message.
We had to manually restart builds and repush all images. I verified that in the container registry the referred images exist - this is the case, there was no change done. Even with images being displayed as avaliable, we had to repush them to get rid of the error.
What happened there?!
Docker is unable to find the image.
$ docker pull eu.gcr.io/seepex-cs/scs-grafana Using default tag: latest latest: Pulling from seepex-cs/scs-grafana 9d48c3bd43c5: Already exists 4842084dac50: Already exists 7cbaa73b9ead: Already exists 9a7207a7a1b5: Already exists 6bb6df97bf66: Already exists e9c24addd21e: Already exists 21ae065ef1d6: Already exists error pulling image configuration: unknown blob
Image is listed in google container registry: