When running following command for cluster down in Kubernetes, I am getting following error:

KUBERNETES_PROVIDER=ubuntu ./kube-down.sh

rm: cannot remove ‘/var/lib/kubelet/pods/16981b98-a3bb-11e5-99fb-00505622b20d/volumes/kubernetes.io~secret/default-token-0i2n6’: Device or resource busy

I tried to remove it forcefully but then also its not getting removed.


I think this was fixed by https://github.com/kubernetes/kubernetes/pull/17315 You might want to check if you have the latest cluster/ubuntu/* files.


In case you're wondering what the underlying mechanics of such an error are, seeing a Device or resource busy typically means there's a kernel-held "lock" on something at that point in the file system and the operation for unlink-ing (that is the syscall that the rm command uses to request this action from the kernel) and is prevented, with an error condition (EBUSY). You can learn a lot more about what syscalls are used by running a command through strace if you want to see all syscalls involved (including the return codes from syscall invocations).

That path looks to me where kubelet manages pod-specific "filesystem-esque" parameters, in this case, what is happening is that every namespace has a default service account that then gets a default token, and that token is transparently mounted inside the pod container's file system. That path mentioned in the error, is the host-level filesystem location of the pod's "internal" filesystem. So in this case, the "lock" in question is a mount-point.

Basically that script is trying to removing that path (perhaps clean-up after itself) and it's still in-use (perhaps the pod was still running), and the script or some command it's invoking is not doing proper error-checking/handling, so you end up seeing that error instead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.