0

We experienced a major outage with our production systems today. Without any user interaction most of the kubernetes pods went down with a "ImagePullBackOff" error message.

We had to manually restart builds and repush all images. I verified that in the container registry the referred images exist - this is the case, there was no change done. Even with images being displayed as avaliable, we had to repush them to get rid of the error.

What happened there?!

Example error

EDIT
Docker is unable to find the image.

$ docker pull eu.gcr.io/seepex-cs/scs-grafana
Using default tag: latest
latest: Pulling from seepex-cs/scs-grafana
9d48c3bd43c5: Already exists 
4842084dac50: Already exists 
7cbaa73b9ead: Already exists 
9a7207a7a1b5: Already exists 
6bb6df97bf66: Already exists 
e9c24addd21e: Already exists 
21ae065ef1d6: Already exists 
error pulling image configuration: unknown blob

Image is listed in google container registry:

enter image description here

Improve this question
1
  • You might need to describe the pod to see an actual reason why it cannot pull an image. You might also need to check the troubleshooting guide about ImagePullBackOff to get more insight on this. Similar situation discussed here as well. I hope this helps
    – Digil
    Nov 5, 2019 at 18:50

1 Answer 1

Reset to default
0

I figured it out.

Since I wanted to save space and money, i´ve setup cleanup jobs for the eu.artifacts.project.appspot.com bucket together with other buckets. This is the bucket where the registry artifacts are stored. After 30 days some of the less active services were auto-deleted...

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .