1

I want to configure RAID for a KVM/LXC server (it's running proxmox as a hypervisor).

This server is now running on 3 SSDs in this configuration:

  • 10GB host root partition + Btrfs partition for some containers
  • LVM-Thin (vg 0)
  • LVM-Thin (vg 1)

I wanted to change this configuration to RAID5 for fault tolerance.

But I can't really tell which configuration is better:

  1. mdadm RAID5 + LVM (and LVM-Thin) on top of that
  2. ZFS RAIDZ (write-hole protection)

This server will be used for hosting a few game savers, websites and databases.

(most of this stuff will be run in LXC containers rather than KVM VMs).

My primary concern is about ZFS eating too much of CPU power, I'm also not sure about using ZFS as underlying storage for VMs.

3

My primary concern is about ZFS eating too much of CPU power, I'm also not sure about using ZFS as underlying storage for VMs.

Checksumming and LZ4 compression are almost free on modern CPUs. If you have constant writes / reads at NVMe speeds it might start to become significant but I doubt it matters for your usecase.

For containters it's great because you can have instant zero-overhead snapshots. For VMs you use zvols - that also provide instant snapshots, however zvols are a mixed bag on Linux accourding to some sources. Personally I've had no problems.

Overall mdadm + LVM might be faster, but ZFS has compression and better snapshots. Just benchmark.

0

If this is a production system, I would stay away from ZFS unless you have experience administering it. It's not as simple as setting it up once and walking away from it.

Using ZFS as storage for your VMs is fine, for all the reasons laid out in mt_'s answer, but it does have some administrative complexities which aren't immediately obvious.

ZFS's compression is Free*, as is encryption as long as you've got a relatively modern CPU. The only major drain on resources that you'll need to really plan for is if you're doing dedupe. If you're going to be dedupilcating data you'll need to throw in a couple GB of RAM for every TB of space in the ZPool.

You're also going to need to think about how you'll expand the architecture in the future, as you can't just add a new disk to the ZPool. This is supposed to be changing in a near-term update to ZFS, but it's been coming soon for ages.

  • Yes, deduplication is a resource hog. I'm running a homebuilt nas with an 8 disk raidz2 with an old Athlon II X2 235e with compression on, but encryption and dedupe off. I can flood the 1gig nic in that box, averaging 950mbps+ reading or writing a single large file, as they are 5.4~7.2k rpm HDD and not SSDs. I didn't try encryption, but dedupe made it run like a dog with barley 1mbps, and/or just freeze. Seeing as it had some of its 8GB RAM spare with dedupe off, I also run a couple of tiny tiny VMs on it too. – BeowulfNode42 Oct 22 '18 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.