6

I recently came across a recommendation from Microsoft regarding the configuration of Logical Volume Management (LVM) in cloud environments, and I find it somewhat puzzling. The recommendation states that while it's possible to configure LVM on any disk attached to a virtual machine (VM), most cloud images won't have LVM configured on the OS disk by default. The reasoning provided is to prevent issues with duplicate volume groups if the OS disk is ever attached to another VM of the same distribution and type, particularly during recovery scenarios. Consequently, the recommendation suggests using LVM only on data disks.

However, I'm struggling to understand the practicality of attaching the OS disk to another VM, especially considering that troubleshooting and diagnostic tasks can easily be performed on separate VM instances. In the event of data recovery, spinning up a new VM without LVM partitions and mounting the LVM volume seems like a straightforward solution.

Furthermore, the argument for using standard partitions over LVM due to their rigidity doesn't seem convincing. LVM offers flexibility in managing disk space, allowing for easier expansion of volumes compared to standard partitions, which can be particularly beneficial if the volume is between other partitions.

I'm curious if anyone can provide insight into scenarios where attaching the OS disk to another VM in a cloud environment would be necessary or advantageous. In my experience, separating system and data disks and attaching data disks to new VMs has been a more practical approach.

I'd appreciate any insights or experiences others might have on this topic.

1
  • In a broader context, it wasn't until ~recently that Microsoft supported Windows volume encryption in Azure. Or SecureBoot. Attaching (attacking?) OS disk is probably more common on Windows than it should be. Microsoft actually has a sophisticated product to protect/prevent it (shielding), another product that will never be supported in Azure.
    – Greg Askew
    Apr 26 at 1:46

6 Answers 6

11

TL;DR: LVM it's redundant in a VM.

The reasoning provided is to prevent issues with duplicate volume groups if the OS disk is ever attached to another VM of the same distribution and type, particularly during recovery scenarios.

This really looks like a cargo cult reasoning. I had a few cases when we moved a virtual disk from one VM to another, but that was always a data disk; for instance, that was our upgrade path: we prepared the new version of a VM with a dummy data disk, and when it was ready, just plugged the real data disk into it from older VM.

I can hardly remember having a VM screwed so badly so it won't boot. Anyway, in such cases, recovery from a backup or complete reprovisioning of the VM will be certainly more appropriate than mounting a system disk to another VM to fix something.

Furthermore, the argument for using standard partitions over LVM due to their rigidity doesn't seem convincing. LVM offers flexibility in managing disk space, allowing for easier expansion of volumes compared to standard partitions, which can be particularly beneficial if the volume is between other partitions.

When you are running in a VM, you already have some volume management, external to the VM. Disks presented to it are already virtual and very flexible; you don't need to waste any resources to support additional, redundant facility. (Not only a runtime translation layer is wasteful, but the need to monitor and manage it within a VM takes resources too.)

Consider this a continuation of the idea that you don't build e.g. software RAID and multipath within VM, but on the host; now, you don't build even volume management within VM, but on the host.

Consequently, the disk usage style for VMs tends to be different than for bare metal servers. Instead of adding one or a few big disks and partitioning them in the VM, you add each potential volume as an individual disk containing at most one partition. When you need to extend space, you increase the virtual disk in the virtualization environment, and then grow a partition and FS in a VM. This is now your LVM.

In most cases you will need a root FS and data FS, e.g. two virtual disks, and chances are the root FS disk of ~30 GB will never need to be resized. Databases often suggest using separate storage for write ahead logs; that will require addition of a dedicated virtual disk anyway, so to place it onto separate storage in the virtualization environment.

There were a few occasions I used more than four virtual disks, but all those cases were degenerate, resulting from someone's awful planning, and eventually such systems were refactored with fewer disks.

3
  • What you call ‘cargo cult reasoning’ is actually an issue for LVM, see my answer for an explanation of why. Apr 24 at 3:57
  • Things get much less black and white with different filesystems that are their own volume management (btrfs and zfs spring to mind) and I would urge people to consider their options on a base by case basis. A simple counter-example may be you had a LUN on a SAN which is now being exposed to a VM rather than a bare-metal system, may already have LVM on it. Overall though I agree with this sentiment -- dont add more cogs to your setup than necessary. Apr 24 at 18:46
  • I would stand against ZFS or something like this within a VM as well. It's a good thing do be used as a SAN backend or on the host, but not in the VM, for the same reason. Apr 25 at 4:48
3

The recommendation states that while it's possible to configure LVM on any disk attached to a virtual machine (VM), most cloud images won't have LVM configured on the OS disk by default. The reasoning provided is to prevent issues with duplicate volume groups if the OS disk is ever attached to another VM of the same distribution and type, particularly during recovery scenarios. Consequently, the recommendation suggests using LVM only on data disks.

If you create two VMs from the same base image, and that base image is using LVM, and then you attach a disk covered by that base image from one to the other:

  • Any PVs on that disk will have the same UUID as the corresponding PVs on the existing disk in the system. PVs with duplicate UUIDs will cause issues during discovery of VGs, because LVM intentionally only uses UUIDs internally for identifying PVs (this is done so that you don’t have to worry about what device node the PV shows up as), and it won’t know which PV to use for a given VG from among those with the same UUID.
  • Any VGs on that disk will have the same UUID and name as the corresponding VGs on the existing disk in the system. This, depending on LVM version and configuration, will result in them either being merged, or both refusing to activate, or possibly only the first discovered activating.

The same issue can also happen with regular systems too, but is much less common unless you’re doing something like cloning systems regularly.

It can also be an issue at the filesystem level (duplicate UUIDs for filesystems can do nasty things), but that’s usually only an issue at boot time unless you’re using BTRFS, and it’s easy enough to work around in most cases because you can choose to identify filesystems by label or device instead.

Furthermore, the argument for using standard partitions over LVM due to their rigidity doesn't seem convincing. LVM offers flexibility in managing disk space, allowing for easier expansion of volumes compared to standard partitions, which can be particularly beneficial if the volume is between other partitions.

How often have you actually needed to resize a partition though? The norm for a VM is not dozens of partitions on one giant disk, it’s one to three on the system disk (firmware boot volume if required, optional /boot, and the root partition), and any data volumes are separate, dedicated disks.

Because the norm for everything but the system disk is one partition per disk, LVM has little to no value anywhere but the system disk (because to resize those partitions, you inherently need to resize the disk itself, which is usually easy on VM setups). And for the system disk, it’s rather rare that you need to resize any of the partitions (the firmware boot volume is generally fixed size, /boot is maybe 1G at most in most cases, and the root partition will be the rest off the disk), so there’s little value there.

3
  • Any decent image will randomise things that are meant to be unique on first boot. If your cloud image uses LVM, then on first boot it should randomise the UUID.
    – abligh
    Apr 26 at 7:42
  • @abligh Except you can’t change the UUID of a PV that’s in use. And if you’re using LVM for your root volume, by definition there will always be at least one PV in use while the system is running, so you can’t work around the UUID collision issue that way. Apr 26 at 14:01
  • @AustinHemelgarn you can change it from the initrd while root is not mounted.
    – abligh
    Apr 26 at 15:33
2

I'm not specifically familiar with the processs on Azure, but with qemu and on AWS, the process of resizing a volume is much simpler than setting up a PV/LV.

scenarios where attaching the OS disk to another VM in a cloud environment would be necessary

Because the /boot is messed up? Or single user mode is password protected and you don't have the password.

2

Short answer: when dealing with storage, the simpler the better. Microsoft recommendations should be read keeping this in mind, especially considering that they addresses a broad audience.

Long answer: while using plain partition is simpler, one should carefully think about partition layout then. The root partition should be the last one, as having other partitions after it means that added disk space will not be immediately usable by the root filesystem. For example, expanding a disk having a swap partition at the end means that any added space will be contiguous to the swap, not to the root filesystem. To expand the latter you then have to remove or relocate the swap partition.

For this reason, and for keeping things simple, I generally do not use LVM when I can get away with a single partition/filesystem for the root disk (with the possible exception of a dedicated boot partition), but I use LVM for more complex setup.

Note that for data disks LVM has some key advantages as thin-provisioning, cache, snapshots, deduplication, etc. While some are provided by the hypervisor/cloud platform also, LVM remains a very useful tools for data volume management.

The recommendation states that while it's possible to configure LVM on any disk attached to a virtual machine (VM), most cloud images won't have LVM configured on the OS disk by default

The plain partitions setup is the default Ubuntu (and other Debian-based distros) setting, which I think is the main motivation behind the statement above. Ubuntu also use a swapfile (rather than a swap partition) by default. Please note that RHEL-based distros uses LVM (and a dedicated swap volume) by default, instead.

The reasoning provided is to prevent issues with duplicate volume groups if the OS disk is ever attached to another VM of the same distribution and type, particularly during recovery scenarios

While true, this is hardly an insurmountable issue: one can identify/activate any PV/VG by its UUID and rename it as needed, or simply use a plain-partition machine to mount and diagnose the problematic root filesystem. Still, that means additional complexity during recovery.

1
  • I remember as I learnt that Microsoft has developed commodore basic. It is a hard experience even since then. But it is much hard to accept: to me, the way from the c64 into the linux was very direct and consequent, but looks not for others. I believe I am more an open source fan as an once 64er.
    – peterh
    Apr 25 at 23:04
0

There is a use case where "attaching the OS disk to another VM in a cloud environment would be necessary or advantageous." Occasionally, people misconfigure their VM in such a way that they don't gain access to it. The normal resolution would be to delete the VM and try again, but sometimes this is not an option.

For this, OpenStack has a "rescue mode". And indeed it works by attaching the operating system disk of the rescued VM to a known-working other VM. In this state you can examine the broken VM's disk and attempt to fix its configuration. Often this will mean creating a way to log in from the console, using a password, or if the network configuration is not the issue to fix, adding an ssh key to some user.

0

If you have LVM on your virtual disks, that extends your administrative capabilities, without the interaction of the host provider. That decreases the expected value of the sum you spend by them; it also decreases your dependence from them.

I believe, this is the major reason.

Another minor reason is not against LVM, but against to rent block devices et all. A block device is a device of disk blocks:

  • it has some administrative overhead to see, what is on them;
  • it is easier to encrypt on way not even the host provider can decrypt;
  • Compared to directory services, there is lesser room for overcommit.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .