Which docker storage driver for new server with NVMe drives?

Question

I am building new server for GitLab CI jobs.

Currently we already have one home-made.

It's using classic HDDs + SSDs for caching so I chose ZFS storage driver which is not performing optimally.

New server will consist only of two NVMe SSDs so I don't need any kind of caching.

I would like to use mirroring for root partition and also for docker volumes partition.

I would like to use stripping for images and containers partitions as it can increase speed and doubles the capacity. All the containers and images are ephemeral anyway.

I read a lot of studies for docker storage performance and every study shows something different.

Most of the studies result in using overlay2, but I would have to use mdadm and have no snapshots possibility.

I was really interested in devicemapper so I could use lvm so I could use snapshotting and thin provisioning. Most of the studies measured low performance with devicemapper driver so that's probably not an option anymore. Maybe I could use overlay2 with lvm volumes?

Last option is ZFS which has no advantages for this server as there will be only two NVMe drives. Also some of the studies measured high and some low performance so I don't know what to think about it.

And also one question. I can use multiple storage drivers(same drivers, but different partitions/volumes to use), right? One for stripped volumes and one for mirrored volumes.

Thanks for answers!

Answer 1

If you are using CentOS/RHEL, stick with overlayfs + XFS, which is the only supported configuration. Note that you can use snapshots in this configuration: all you need is to use XFS above a thin LVM volume (but you must monitor it to avoid out-of-space conditions). For better thin volume performance, remember to disable zeroing and to use an appropriate chunk size (512KB+ is a good baseline, unless you deeply care about snapshot space efficiency).

Finally, be sure to create the XFS filesystem with the -n ftype=1 option (recent mkfs.xfs should do it automatically, but please double-check it). From RH docs:

XFS file systems must be created with the -n ftype=1 option enabled for use as an overlay.