3

For some use cases I'd like to have the possibility to create a volume with docker volume create, and fill it with data. Then I want to create a new volume that is only a copy of the first one, but no data needs to be copied, only changed files should be written.

This is what Docker already does with images and containers using aufs or overlayfs. Do you know any strategy to solve this for volumes? If it does not work with docker volumes right now, can you tell me how to do something like this on the Docker host? (Then I could mount those folders from the host.)

By now I have two use cases for it, maybe you have a completely different idea to solve it:

  1. Very big Git repository (around 10GB): I need to run multiple containers with this repository, some of them also commit changes. So it must be independent from other containers, push commits to the remote repository and then the copied volume may be deleted.

  2. Very big mysql databases (around 130GB): For testing purposes and other migration tasks I'd like to not have to copy the whole database.

1 Answer 1

4

Given the lack of other answers to this, I'll chime in. The best way to achieve this today is to do this yourself with overlayfs. It's included in the Linux kernel nowadays and so you can get going with no additional packages or modules (provided you're on 3.18 or newer).

Using overlayfs requires a working directory, an 'upper' directory and a mountpoint in addition to your existing directory that you don't want to change.

Assuming I have data in myreadonlydir which I don't want containers to change, I can create two overlays like this:

mkdir upperdir1 workdir1 mountpoint1 upperdir2 workdir2 mountpoint2
sudo mount -t overlay overlay -o lowerdir=myreadonlydir,upperdir=upperdir1,workdir=workdir1 mountpoint1
sudo mount -t overlay overlay -o lowerdir=myreadonlydir,upperdir=upperdir2,workdir=workdir2 mountpoint2

Now I have two mountpoints, mountpoint1 and mountpoint2 which I can attach to my containers, e.g.

docker run -v $PWD/mountpoint1:/data alpine:3.6   # first container
docker run -v $PWD/mountpoint2:/data alpine:3.6   # second container

And each container will see the contents of myreadonlydir inside /data. When they write to anything in /data (delete files, append to files, create files, whatever), overlayfs will only 'write' to the upperdir, and the data in myreadonlydir is left alone.

Since the containers are using different mountpoints, each container's modifications to /data do not affect the other container.

If you want to be really safe, you can mount your data -o ro to protect it from any writes.

You could quite easily throw together a script to do this for each container you spin up in your test environment.

Note though, I've no idea how well overlayfs would perform when using it with a big MySQL database... under the hood it performs 'copy-up' operations inside the workdir when a file is written. Ideally your workdir should be on fast storage - SSD or a tmpfs if you've enough RAM.

More info @ https://wiki.archlinux.org/index.php/Overlay_filesystem.

2
  • Does this really work? I can mount an overlay directory successfully into the docker container, but inside docker, the directory is empty.
    – weynhamz
    Oct 10, 2020 at 10:36
  • OK, this works in Global Docker, but not in rootless Docker.
    – weynhamz
    Oct 10, 2020 at 14:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .