I have this weird error on my Kubernetes test cluster. I'm running Kubernetes 1.7.0 and docker Docker version 1.12.6, build 78d1802
I have a separate volume mounted as /dockerdata and symlinked /var/lib/docker into this volume. I have ample space available on both my system volume and the dockerdata volume (more than 50% free). I still get ImagePullBackOff errors from kubernetes, the detailed error is:
Failed to pull image "<redacted>": rpc error: code = 2 desc = failed to register layer: ApplyLayer exit status 1 stdout: stderr: open /usr/share/man/es/man1/fakeroot-sysv.1.gz: no space left on device.
If I manually clean up by removing stale images I can get around this, but as far as I can see I shouldn't have to - since none of my volumes are nearing full capacity (which I guess is also why the Kubernetes GC doesn't kick in).
I'm guessing there's something I don't understand about how Docker works in relation to local disks but I'm stumped. Any pointers appreciated.
Here's some more info: symlink info:
file /var/lib/docker
/var/lib/docker: symbolic link to /dockerdata
df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 2014736 0 2014736 0% /dev
tmpfs 404520 44628 359892 12% /run
/dev/xvda1 20263528 4121240 16125904 21% /
tmpfs 2022600 0 2022600 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 2022600 0 2022600 0% /sys/fs/cgroup
/dev/xvdb 51475068 18385324 30451920 38% /dockerdata
tmpfs 404520 0 404520 0% /run/user/1000
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
