I am experiencing something strange while integrating promtail to my stack

ERROR: for asmodius_promtail  Cannot start service promtail: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/asmodius_nginx-logs/_data\\\" to rootfs \\\"/var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged\\\" at \\\"/var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged/var/log/nginx\\\" caused \\\"mkdir /var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged/var/log/nginx: read-only file system\\\"\"": unknown

The volume mentioned in this error is the one I use so promtail could read Nginx logs. (It only reads the files and does not need to write anything in the dir) Here is a portion of the docker-compose.yml

version: '3.7'
    container_name: ${SERVER_NAME}_nginx
    restart: always
      - monitoring
      - web
    image: nginx:1.17.5-alpine
    entrypoint: /entrypoint.sh
      - 80:80
      - 443:443
      - ./nginx/site.conf:/etc/nginx/conf.d/site.conf
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/entrypoint.sh:/entrypoint.sh
      - nginx-logs:/var/log/nginx
      - certificates:/etc/letsencrypt/

    restart: always
    container_name: ${SERVER_NAME}_promtail
    image: grafana/promtail:master-b652f0a
    command: -config.file=/etc/promtail/docker-config.yaml
      - nginx-logs:/var/log/nginx
      - /var/log:/var/log:ro
      - ./promtail/docker-config.yaml:/etc/promtail/docker-config.yaml:ro
      - monitoring

and the volume "inspection"

        "CreatedAt": "2020-04-28T20:08:41Z",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "asmodius",
            "com.docker.compose.version": "1.24.1",
            "com.docker.compose.volume": "nginx-logs"
        "Mountpoint": "/var/lib/docker/volumes/asmodius_nginx-logs/_data",
        "Name": "asmodius_nginx-logs",
        "Options": null,
        "Scope": "local"

Strangely enough, the very same docker-compose.yml works like a charm on my Debian 9

  • Docker version 19.03.8, build afacb8b7f0
  • docker-compose version 1.22.0, build f46880fe

but fails on CentOS

  • Docker version 19.03.8, build afacb8b
  • docker-compose version 1.24.1, build 4667896b


I don't quite believe that docker-compose version could be the problem. I tried the following

  • stop the entire stack, rm nginx-logs volume and start again
  • use :ro not use :ro


Since I do not entirely understand the ERROR message, an explanation of the following elements might help me figure what I am doing wrong.

  • read-only file system\\\"\"": unknown: I just don't get what "\" is
  • why does it mention a merge ? A merge with what?
  • mkdir.... so it tries to create a folder somewhere in /var/lib/docker. But why does it have to do so for this specific volume while all the others (and there is quite a lot of them) succeed every time I docker-compose up -d --force-recreate (for it is how I add/update services on this machine)

All insights are welcome

2 Answers 2

mkdir /var/lib/docker/overlay2/f6e22c4d99f1e9907a8ff4b05338386a38b9a7cd802944a877102bb805b97dac/merged/var/log/nginx: read-only file system\\\"\"

This is complaining that it cannot create a directory under /var/lib/docker. None of that filesystem should be read-only. Perhaps you didn't start the docker engine as root, or perhaps there's something wrong with the filesystem itself. I'd double check that you haven't run out of capacity, either in free bytes or free inodes, with:

df /var/lib/docker/.
df -i /var/lib/docker/.

If you are running Docker Desktop, then as a last resort, destroy and recreate the docker VM. This will result in losing all the volume data, built images, running cntainers, etc, so backup anything you want to save.


This might be a bit late but I battled with similar issue:

I believe the issue is caused by having two (container-internal) mount locations being overlaid but simultaneously having differing 'read-only' and 'read-write' permissions.

In the original post, you'll notice that for promtail service has these two mounts:

  - nginx-logs:/var/log/nginx
  - /var/log:/var/log:ro

here the first mount is mount is read-write, but second is read-only.

Note that

  1. from the container's perspective the /var/log/nginx is inside /var/log/.
  2. If the mount path inside the container (and/or volume) does not exist, docker will attempt to create it
  3. it seems that docker will attempt to mount the paths from "top to bottom" from directory hierarchy perspective, not in the order they were listed in the docker-compose.yml.

This means that /var/log/ will be mounted first, /var/log/nginx will be attempted second. If the nginx subdirectory does not already exist (very likely), then obviously it cannot be created if /var/log is marked read-only.

The error message attempts to inform user of this, but in rather confusing, un-intuitive way.

The easiest solution is to not mark parent directories (inside container) read-only if any children are intended to have write permission.

Another solution is to ensure that /var/log/nginx path exists inside the backing medium (the nginx-logs volume in this case) before the service is started. However, this is rather cumbersome to ensure reliably, e.g in cases where you recreate the volume or move hosts.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .