6

So, I've got a pretty awesome KVM setup running now, CentOS5.5 on the host and guest domains, libvirt managing all the configuration, etc. The guest domain filesystems are stored in LVM on top of a hardware RAID5 volume, so I have flexibility for backups and low-level data redundancy.

I tested out virt-clone today, and it worked amazingly well, except that it took about 30 minutes to replicate 24G of data from the suspended domain's LVM disks to the new LVM volumes for the new virtual machine.

My question is: couldn't I just use an LVM snapshot to create the new VM's root disk? For example: lvcreate -s guest1_root -n guest2_root -L 8G raid_vg

Now, my understanding of LVM snapshots is that the snapshot stores an inverse delta of the changes made to the original blocks so that the snapshot takes up little actual space and the original blocks can be read out even after the original volume has been written to. LVM2 adds read-write snapshots, which opens up this interesting possibility.

Indeed, the LVM HOWTO even suggests using this feature in conjunction with Xen:

This opens up many new possibilities that were not possible with LVM1's read-only snapshots. (...) It is also useful for creating volumes for use with Xen. You can create a disk image, then snapshot it and modify the snapshot for a particular domU instance. You can then create another snapshot of the original volume, and modify that one for a different domU instance. Since the only storage used by a snapshot is blocks that were changed on the origin or the snapshot, the majority of the volume is shared by the domU's.

This seems like a really powerful tool, and I'm wondering if anyone has tried it in their production virtualization environment, can think of any vendors (Citrix, VMWare) that take this approach, or can think of any serious problems with this idea. I can think of the following potential problems:

  • Potential filesystem problems if the "original" guest is running at snapshot time.
  • Host domain LVM performance as the "inverse delta" and "written block" logs are appended to. It may be very fast, but I don't know, and this seems like a weird idea, so it bears testing and benchmarking.
  • More eggs in one basket: if the "original" LVM volume gets corrupted, then the snapshotted volumes are hosed as well. Of course, RAID5 mitigates disk corruption, but it's worth noting.
  • Can you snapshot a snapshot? I assume so.

As people who likely have more virtualization experience than I, is there anything that makes want to run screaming from such a setup?

4 Answers 4

4

It's perfectly fine to do this. What you do not want is to have the parent of the snapshot (the original, or the source, or whatever you want to call it) to be in use at the same time, because it will cause IO multiplication (Hubert was right about this, it is just easy to prevent by not using the source volume all the time).

If you have one master OS install on an LVM, and you snapshot that four times, you will not have much of an IO penalty, since you are only writing to the individual snapshot volumes. Of course, it is not free, but neither are other forms of snapshotting on other filesystems or virtual disks. There is always a cost somewhere.

One other thing Hubert is right about, is that you have to think about the sizing of your snapshots. You will want to make sure the snapshot volumes are able to keep writing. A full snapshot volume will break stuff badly. A failsafe way to prevent this, is to make the snapshot volume the same size (or bigger) than the source volume. You loose the benefit of using less diskspace this way, though.

You know that qemu images are snapshot capable too?

2
  • Didn't know about qemu images, that's cool. Thanks for the detailed reply; good to know that LVM images can work.
    – phred
    Oct 1, 2010 at 17:19
  • 1
    Also, other answerers, ZFS isn't the panacea you think it is. It's a great filesystem, but it's not and never will be a networked/clustered filesystem, and there are inherent synchronization problems with trying to share a Z filesystem between multiple VMs, just like ext3, just like reiser, etc.
    – phred
    Oct 1, 2010 at 17:36
4

While I didn't try LVM for KVM storage, I did use it for shadow volume feature of samba and I can tell you one thing: the performance was abysmal.

Each and every snapshot requires an additional write to happen. If you have one base snapshoted volume and 4 snapshots, the amount of writes that goes to the drives is multiplied by 5 when you write to base volume.

As for your specific questions:

  • LVM freezes a filesystem while the snapshot takes place (stops writes, flushes cache, does the snapshot, resumes writes)
  • as I said, it's very slow
  • Yes, corrupted base volume renders all snapshots unusable, what's more, if you run out of space allocated for the snapshot deltas, the snapshot is hosed as well
  • yes, you can snapshot a snapshot

Unfortunately, there are only 3 systems I know of that work well with snapshots: NetApp WAFL, ZFS and btrfs. If the system is not critical btrfs could be worth a shot.

4
  • 1
    LVM snapshots work fine. I use them all the time. Your setup is just flawed. LVM snapshots are not meant for this. What works great for example, is to snapshot a MySQL data volume, back that up and then destroy the snapshot again. Having four snapshots around all of the time and then writing to the source: of course performance will be bad: you are quadrupling IO.
    – wzzrd
    Sep 30, 2010 at 9:33
  • Yes, LVM snapshots for backups are a good idea and work well (I use them all the time), but having them all the time trashes performance Sep 30, 2010 at 9:47
  • Yeah, LVM snapshots work amazingly well for MySQL for backups, agreed. Understood that the "source" of the snapshots needs to be quiescent to not totally kill performance, since yeah, it'd have to write the "negative blocks" back to all of the snapshotted volumes. ZFS snapshots wouldn't help in this circumstance, since ZFS "cannot provide concurrent access from multiple hosts" (according to Wikipedia), which is possible with LVM so long as the hosts all share a common hypervisor.
    – phred
    Oct 1, 2010 at 17:23
  • you can still create filesystem images on ZFS, this way ZFS doesn't differ in any way from raw LVM Oct 1, 2010 at 23:03
2

Thin-provisioned LVM should be considered a prime option for this scenario, here in 2019.

Thin LV performance is good, and they function like separate volumes so once a snapshot is made you don't have to worry about the care and integrity of the original (it can be damaged, deleted, etc. without affecting the snapshot).

The OP's concern for "the snapshot takes up little actual space" isn't really satisfied by traditional LVM since space must be pre-allocated in a monolithic fashion for each snapshot. But Thin LVs are allocated like sparse files, and actually do take up little space.

The trade-off for thin provisioning is that the available space in the thinpool must be monitored just like a filesystem to avoid filling it up. Linux distros typically have daemons to monitor this and send warnings or take action when a thinpool reaches a near-full state.

0

I would go so far as to say that ZFS could be worth a shot, its possible to setup a Nexenta based system backed by some kind of disk, either fiber, iscsi, etc, fairly easily, and has pretty good performance. I would recommend this approach over local storage if performance is not absolutely critical any day since this would give you a simple recovery scenario if your virtualization server went down.

3
  • 1
    Except for the fact that you need an extra system...
    – wzzrd
    Sep 30, 2010 at 9:29
  • Yes, and once I have my Nexenta system (had to look it up: OpenSolaris kernel + Linux-y userland tools) I still have to export the filesystem to my Linux virtualization server, which means using NFS or CIFS, which has performance implications and won't allow me to do the operation I need, which is: stop VM from writing to the filesystem and snapshot the file on disk. Since ZFS isn't a network protocol, this means I need to script up some way to have the virtualization server run zfs snapshot on the fileserver, but that sounds like a nasty little house of cards. No thank you.
    – phred
    Oct 1, 2010 at 17:34
  • I don't consider having an extra system a bad thing, I guess it really depends on how many vms you are dealing with and what they are being used for. There are lots of ways to skin this cat. I generally think separating your storage from your server is a good thing in most cases even if its only at the fiber channel / iscsi level versus another layer of indirection that nexenta or some kind of managed filesystem platform would provide. Phred brings up very valid concerns, again implementation specific. For future reference there is a Nexenta API, it's relatively straight forward.
    – MattyB
    Oct 3, 2010 at 17:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .