I'm fairly new to k8s and minikube, and could use help understanding the frequent-but-not-ubiquitous error I get when running kubectl logs -f <POD NAME>

I get this, sometimes, and unpredictably, on pods in my own namespace, and in kube-system, e.g. when trying to tail the ingress controller logs. There is no apparent error in the pods themselves, and the output of the kubectl logs command contains normal-looking log lines right up to, e.g. failed to watch file "/var/log/pods/kube-system_nginx-ingress-controller-586cdc477c-95pgh_4b8310a4-5f9b-11e9-9666-0800270e7244/nginx-ingress-controller/1.log": no space left on device$

What's the ultimate source of this, and how do I resolve it?


It sounds like the node has run out of inotify watches.

You can check or set the number of available watches using the sysctl fs.inotify.max_user_watches on each Kubernetes node.

You will probably find it has been set to some ridiculously low number; by default it is autotuned based on the amount of memory the system has.

[root@small ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

[root@large ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 1048576

Whatever the amount is currently, double it until the problem stops occurring.

(Use minikube ssh to access a Minikube node.)

| improve this answer | |
  • 1
    hey thanks! for those like me, new to k8s sysctl the command to set it is sudo sysctl fs.inotify.max_user_watches=1048576 – Ben Apr 18 '19 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.