3

Should I use ZFS inside a virtual machine? According to existing information (in particular this thread, which I will quote below) it is not a good idea, unless certain measures are provided (such as using VT-d). The basic problem allegedly is that the hypervisor may report a block to be written when in fact, it has not yet been written. Allegedly, ZFS is particularly sensitive to such a discrepancy, compared to other filesystems:

The specifics of why it's so bad relate to it's fsck-less design.

It was also acknowledged in that discussion that:

The only scenario where "pointer written before data" turns into "pointer written but no data" is still the event of a crash of either OS or the hypervisor, as far as I can imagine.

But what is the worst-case consequence in case of such a crash?

  • Corruption or loss of files written around the time of the crash?
  • Corruption or loss of any other file in the dataset?
  • Corruption or loss of any file in a dataset that was not being modified around the time of the crash, e.g., a snapshot that was taken long before the crash?

Is there any silent corruption possible, which cannot be detected by a scrub?

I have tried to simulate a crash by using VirtualBox and selecting Close / Power Off while data was written to a ZFS inside of a VM. I did this about 20 times, but could not detect any problems. Maybe this method of testing is not appropriate. Are there better ways to experiment with this?

  • 1
    Do you care to describe the virtualization technology and environment you're planning? – ewwhite Sep 23 at 8:06
  • The only situation this matters is when you make snapshots for CBT and similar based backup and at the moment of backup there is intense writing. However, even so it depends of the proper or improper configuration of the ZFS guest. – Overmind Sep 23 at 8:24
  • @ewwhite The relevant VM is provided to me by our computing center. They are running a vSphere Cluster, and in the web console, it says: "Compatibility: ESXi 6.5 or higher (VM Version 13)". Can you do an assessment based on this? If you need any additional information about the system, please let me know, and I will try to find it out. – Lasse Kliemann Sep 23 at 11:32
6

Basically any modern hypervisor (VMWare, Xen, KVM, HyperV, even VirtualBox) supports barrier passing: when a VM explicitly flushes something to disk (by issuing a barrier/FUA), the hypervisor will pass the barrier down to the host, forcing the same flushes executed by the guest OS. In other words, no corruption is expected for important/durable writes (as the one used by the filesystem itself to update its metadata).

While most hypervisor can be confiured to ignore flushes, this can jeopardize any filesystems - XFS, EXT4, etc. will be exposed to serious corruptions as much as ZFS.

Back to the main question: it is perfectly safe to use ZFS inside a guest OS, and I have first-hand experience on similar setup. This will enable the guest to use advanced features as compression, snapshots and send/receive. However this can led to somewhat lower guest performance (ZFS is not enginereed to be a benchmark-winning filesystem). Moreover, as many SANs implements the very same features at the disk image level, you should evaluate if the performance impact due to double-CoW is worth the additional flexibility at the guest level.

For the reasons above, I generally use ZFS at the disk-image/hypervisor level, while using XFS or EXT4 inside the virtual machines themselves. However, on scenario where I have no visibility on the underlying SAN/storage (and its snapshot/compression/replication policies), I sometime use ZFS at the guest level.

In these cases, the added features more than compensate the performance impact vs, for example, a plain XFS setup. And I have no stability/durability problems at all.

Side note: VT-d is only useful if you plan to pass the raw disks (or other hardware devices) to the guest itself. If using file or volume based virtual disk, you are not using VT-d.

  • Thanks, this helped a lot. I'm using ZFS exactly for the mentioned benefits (snapshots in particular), and I have no control over the hypervisor setup. ZFS is delightful to work with, and going back to anything else in this situation would have been painful. I will continue using ZFS on the VM. – Lasse Kliemann Sep 23 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.