4

I'm in the process on playing around with KVM and setting up a linux on a linux-server, so I can

  • reboot the (guest)server
  • encrypt the rootFS of the guest without having to do initramfs-tricks to get remote reboot to work.

I'm thinking about using btrfs as filesystem, since it's a single ssd-disk and no RAID.

Following thoughts:

  • If I use btrfs on the host, i have excellent data-integrity, all the checksums etc.
  • So I can use something fast and simple, maybe even ext2 on the guest?

Or should I do it the other way round?

3 Answers 3

8

Running BTRFS on the host for disk images (qcow2, etc) is a really bad idea. The style of disk writes to an image file is the worst IO pattern for btrfs, the tuning KVM page does not contain a lot of tips:

http://www.linux-kvm.org/page/Tuning_KVM

But on this one they are clear:

"Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest."

I use ext4 for local files and a nfs store via ZFS on solaris for remote disk images. We will be switching to zfs based iscsi shares in the near future. If you don't need a nas / central file store and are going to have a relatively stable number of VMs DukeLion's suggestion is best.

We use the cluster for software testing, so we are creating 1000s of VMs each day using qcow2 overlay files. LVM does not support doing that.

2
  • A follow-up question: When using encryption, should i use raw-crypt-lvm-diskImage, or raw-lvm-crypt diskImage or raw-lvm-diskImage-crypt. I don't care if all vms share one password, i merely ask because of performance etc. May 31, 2012 at 11:54
  • I have not needed to do encryption, so can't say a ton about that. We use it for a internal testing lab, no need to encrypt testing VMs.
    – n8whnp
    Jun 25, 2012 at 17:20
6

It all depends on what you want to achieve.

I would not recommend use ext2 for root fs in any server - virtual or physical. You may get slower performance in some cases and filesystem inconsistencies is still possible, making reboots incredibly slow.

If you want simplicity and performance - I'd suggest you to put guest image on lvm logical volume and use btrfs/ext4/xfs in guest.

6
  • and what on the host? May 20, 2012 at 15:14
  • raw lvm logical volume maybe with encryption
    – DukeLion
    May 20, 2012 at 15:16
  • +1 for the LVM approach
    – dyasny
    May 20, 2012 at 16:19
  • Hm, that way sounds really nice, but as far as I know you can to the volume management with btrfs as well. So maybe just btrfs instead of the lvm? May 20, 2012 at 19:02
  • 1
    btrfs is a filesystem approach, having more unneeded features and functionality. LVM is dealing with plain block devices. There's nothing particularly bad with btfs, but with LVM it will be more simple and efficient.
    – DukeLion
    May 21, 2012 at 4:20
0

As of this writing, it's been more than 10 years since the original question. And, I daresay that the answer currently isn't as obvious, as it earlier may have been.

BTRFS has an interesting feature for use on the VM host: snapshots on subvolumes. Effectively, it allows you to create a snapshot of a "directory" (a subvolume, really). A subvolume can contain a number of files.

It is true that, as a COW filesystem, btrfs really suffers a spanking from a VM working with its disk image. That is, unless you disable the COW algorithm. You can disable the COW in two ways:

  1. using the nodatacow mount option, on the whole volume (doesn't work selectively on subvolumes)

  2. using chattr -C . To check the result, use lsattr. If you want this to apply on a subdirectory or a subvolume, apply the "no COW" attribute on the freshly created subdir or subvolume, so that any files newly created would inherit that attribute.

Turning off COW will have the result, that BTRFS will avoid doing the COW trick if at all possible. Only when absolutely necessary, such as when a snapshot gets created and is in use, a minimal amount of COW still happens.

In the guest, I'd use any reliable FS with a maintained codebase, that is preferably not a resource hog. Ext4 sounds like "not a bad option".

To save disk space on the host, you can tell namely QEMU to observe the "discard" or TRIM disk commands coming from the VM guest. In a -drive definition, include discard=unmap,detect-zeroes=unmap . In the VM guest instance, you have a choice:

a) do not use the "discard" option when mounting volumes. Traditionally, with flash SSD drives, the discard/TRIM operation tends to be blocking and lengthy. It harms runtime performance. Thus, since shortly after the discard/TRIM has been implemented in Linux, the advice has been: prefer to steer clear of TRIM, and only run fstrim -av every once in a while manually, when it suits you, i.e. at a time when the system load is low. There have also been some optimizations of the disk IO subsystem, to achieve traffic patterns that result in "TRIM-like behavior and allocation efficiency" without actually using the TRIM/discard command. Two older articles on the topic: 1 , 2 .

b) in case you do not have an SSD underneath, i.e. you disk drives are actually spinning rust, you can still use TRIM/discard in a VM guest to signal to the VM host, that it can release some space in the underlying QCOW2 image. "Punch a hole", literally. The QCOW2 image is sparse, and by releasing unused space properly, you'll save a lot of actual disk capacity.

I don't have a longer-term experience, I have effectively just installed such a system (host + 1 guest, about 55 GB of data in two QCOW2 images) and it seems to work fairly well. While migrating the OS and data from bare metal into the VM, I kept observing the sytem with top and iostat, and the numbers rolling on the screen seemd perfectly plausible.

In the way of optimizations, I am using the noatime mount option in both the host and the guest, and I also like to adjust the dirty_ratio, dirty_background_ratio and dirty_expire_centisecs to increase the "writeback cache". Seems to work for me.

6
  • 1
    BTRFS subvolumes and snapshots are the "technology preview" in Proxmox VE; their ZFS's counterparts are considered "stable". I've used both; both work, but I/O performance of both is mediocre. QCOW2 files on ext4 is not too fast, which is expected, because the structure of the QCOW2 is filesystem-like, and it is itself inside FS, and it hosts an FS, so we have an awful sandwitch of translations. And, for completeness, there could be thin LVM on the host to provide guest disks; in my own experience this gives the best performance (of all the thinly provisioned storage options). Jan 19, 2023 at 6:50
  • @Nikita_Kipriyanov thanks for your informed comment :-) Yes the layering of filesystems certainly implies extra overhead and an amplification of IOps. I am aware of alternative possibilities in terms of LVM, in which I am not at all conversant. Thanks god for alternatives :-) For the freedom of choice. Different backgrounds / needs / motivations may warrant different choices. You can tilt the compromise in this or that direction. I appreciate the finer granularity of the VFS-level subvolumes/snapshots, and I'm not bothered by IO bandwidth, for my particular case.
    – frr
    Jan 19, 2023 at 10:21
  • I really just needed a way to virtualize a single Linux system, running a bloated clutter of open-source software ridden by dependency hell, with no good built-in backup options. The underlying task from the management being "arrange solid regular backups of the darn thing". There probably won't ever be any further VM's on that particular host, and both the host and the guest are running pretty much idle most of the time. Mission almost accomplished :-)
    – frr
    Jan 19, 2023 at 10:27
  • 1
    probably Docker is for you then. Or LXC, or other containerized technologies. Either way you don't run into the need of running the VM and managing a separate virtual disk resource Jan 19, 2023 at 10:31
  • To me, docker doesn't feel opaque enough. I really needed to take the existing setup and lift it into a VM with minimal disturbance or rework
    – frr
    Jan 19, 2023 at 10:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .