0

I've deployed a Gitlab 12.10 instance on microk8s 1.18 on my Ubuntu 19.10 server. I've noticed repeatedly that some of the pods go into status CrashBackoffLoop or Init:CrashBackoffLoop with error messages like the following:

Message: failed to create containerd task: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/var/snap/microk8s/common/var/lib/kubelet/pods/de572f6b-f738-4b1a-ae02-80fef062cabe/volume-subpaths/sidekiq-secrets/dependencies/2\\\" to rootfs \\\"/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/3ae6fc33b02192c3940af4ebc991a47b1fd0afc2533901af50ae5a5f93585d1c/rootfs\\\" at \\\"/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/3ae6fc33b02192c3940af4ebc991a47b1fd0afc2533901af50ae5a5f93585d1c/rootfs/srv/gitlab/config/secrets.yml\\\" caused \\\"no such file or directory\\\"\"": unknown

Unfortunately, I'm not sure if this problem is specific to Gitlab or a general issue with my microk8s cluster. It's not always the same component that crashes, the ones that seem to fail most frequently are gitlab-runner, sidekiq and unicorn, but I've also seen others with a similar "no such file or directory" error from time to time.

When I delete a crashed pod, the new one that gets created starts without problems, then everything seems to work for a while until one or more of the pods crash again.

Any idea what might be causing this?

3
  • Hi Jens, welcome to S.F. You deployed GitLab 12.10 how? Via their supported Helm chart? And are you sure you have enough disk space to run all of GitLab on one Node? – mdaniel May 17 '20 at 3:44
  • Yes, I used their supported Helm chart and they also list some settings for a small deployment. But that only mentions CPUs and RAM, nothing about required disk space. The defaults for the Persistent Volume Claims are around 80GB, which is no problem, but I don't know how much storage is required apart from the PVCs. – Jens May 17 '20 at 5:01
  • You said it works when it gets recreated until another pod fails. Which one is crushing and why, can you post logs from those pods? – Crou May 20 '20 at 11:40
0

I had the same problem; initially, a full server reboot fixed it, but it then returned intermittently.

Finding

Finally, it turned to be a permission issue with the "raw" secrets files (like

/var/snap/microk8s/common/var/lib/kubelet/pods/<pod_guid>/volume-subpaths/task-runner-secrets/task-runner/<file_no>
) for the 3 main pods (sidekick, unicorn & task-runner) - all other secret and config map file entries had
r--r-----
permissions, with
root:<user group>
ownership - but these secret files also had
r--r-----
, but for
<user>:<user group>
, which means "not readable for anyone but <user>" - which sometimes works (if microk8s gets started in the user context) but sometimes fails (no idea about the logic of micro8ks here)

Solution

chown root:<user group>
for these files fixes the issue

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.