1

I have this weird error on my Kubernetes test cluster. I'm running Kubernetes 1.7.0 and docker Docker version 1.12.6, build 78d1802

I have a separate volume mounted as /dockerdata and symlinked /var/lib/docker into this volume. I have ample space available on both my system volume and the dockerdata volume (more than 50% free). I still get ImagePullBackOff errors from kubernetes, the detailed error is:

Failed to pull image "<redacted>": rpc error: code = 2 desc = failed to register layer: ApplyLayer exit status 1 stdout: stderr: open /usr/share/man/es/man1/fakeroot-sysv.1.gz: no space left on device.

If I manually clean up by removing stale images I can get around this, but as far as I can see I shouldn't have to - since none of my volumes are nearing full capacity (which I guess is also why the Kubernetes GC doesn't kick in).

I'm guessing there's something I don't understand about how Docker works in relation to local disks but I'm stumped. Any pointers appreciated.

Here's some more info: symlink info:

file /var/lib/docker /var/lib/docker: symbolic link to /dockerdata

df Filesystem 1K-blocks Used Available Use% Mounted on udev 2014736 0 2014736 0% /dev tmpfs 404520 44628 359892 12% /run /dev/xvda1 20263528 4121240 16125904 21% / tmpfs 2022600 0 2022600 0% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 2022600 0 2022600 0% /sys/fs/cgroup /dev/xvdb 51475068 18385324 30451920 38% /dockerdata tmpfs 404520 0 404520 0% /run/user/1000

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial

2
  • Inodes perhaps? df -ih
    – HoD
    Oct 2, 2017 at 8:09
  • very likely. Seems that Inodes has issues with the overlay storage driver on docker. Overlay2 seems to be supported on Kubernetes 1.8, so I guess we'll upgrade, switch drivers, and hope this goes away.
    – Trondh
    Oct 11, 2017 at 5:34

1 Answer 1

-1

You can try this: ls -lah ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

Also then try to delete this file and will not affect any of your containers.

1
  • Thanks! We haven't had this problem after switching to Overlay2 and Docker 17.03.
    – Trondh
    May 2, 2018 at 21:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.