tail -f not following log file in Docker container

Question

I am seeing some strange behavior running the tail -f command inside a Docker container in CoreOS.

There are a number of variables I can think of that may be contributing to the problem but I am not sure what I need to do to troubleshoot first. On CoreOS I am running the newest version with overlayfs support, as well as a newer version of Docker (1.4.1).

What is interesting is that I am able to tail logs successfully on a different host OS (Ubuntu 14.04) which is running a different version of Docker (1.3).

I can produce strace logs if that helps to troubleshoot, they seem to be significantly different across hosts. For example, on the host that isn't working the strace stops after reading in the following in the strace output:

04:03:03 inotify_add_watch(4, "f017f0a1-a1e9-11e4-90bc-027e0f87cac6-paster.log", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 1 <0.000028>
04:03:03 fstat(3, {st_mode=S_IFREG|0644, st_size=12229, ...}) = 0 <0.000022>
04:03:03 read(4, 0x7711f0, 64)          = ? ERESTARTSYS (To be restarted if SA_RESTART is set) <3.101545>

I am just not familiar enough with strace to be very good at interpreting the results.

Answer 1

Not a solution but a workaround:

I have the same problem on CoreOS 607.0.0 and reproduced this problem in containers based on Ubuntu or Fedora. However containers that use busybox do not have this issue. Two workarounds:

1) use a busybox-based container image such as alpine

2) install busybox in your existing container and run

busybox tail -F <logfile>

Answer 2

This is a bug introduced in CoreOS 561 when the default file system was changed from btrfs to the overlayfs. I found a workaround to basically format the filesystem using btrfs when a host is first started, and that seems to be working for now.

Include the following in your CloudInit:

#cloud-config
coreos:
  units:
    - name: format-var-lib-docker.service
      command: start
      content: |
        [Unit]
        Before=docker.service var-lib-docker.mount
        ConditionPathExists=!/var/lib/docker.btrfs
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/truncate --size=25G /var/lib/docker.btrfs
        ExecStart=/usr/sbin/mkfs.btrfs /var/lib/docker.btrfs
    - name: var-lib-docker.mount
      enable: true
      content: |
        [Unit]
        Before=docker.service
        After=format-var-lib-docker.service
        Requires=format-var-lib-docker.service
        [Install]
        RequiredBy=docker.service
        [Mount]
        What=/var/lib/docker.btrfs
        Where=/var/lib/docker
        Type=btrfs
        Options=loop,discard