Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So, I've got a pretty awesome KVM setup running now, CentOS5.5 on the host and guest domains, libvirt managing all the configuration, etc. The guest domain filesystems are stored in LVM on top of a hardware RAID5 volume, so I have flexibility for backups and low-level data redundancy.

I tested out virt-clone today, and it worked amazingly well, except that it took about 30 minutes to replicate 24G of data from the suspended domain's LVM disks to the new LVM volumes for the new virtual machine.

My question is: couldn't I just use an LVM snapshot to create the new VM's root disk? For example: lvcreate -s guest1_root -n guest2_root -L 8G raid_vg

Now, my understanding of LVM snapshots is that the snapshot stores an inverse delta of the changes made to the original blocks so that the snapshot takes up little actual space and the original blocks can be read out even after the original volume has been written to. LVM2 adds read-write snapshots, which opens up this interesting possibility.

Indeed, the LVM HOWTO even suggests using this feature in conjunction with Xen:

This opens up many new possibilities that were not possible with LVM1's read-only snapshots. (...) It is also useful for creating volumes for use with Xen. You can create a disk image, then snapshot it and modify the snapshot for a particular domU instance. You can then create another snapshot of the original volume, and modify that one for a different domU instance. Since the only storage used by a snapshot is blocks that were changed on the origin or the snapshot, the majority of the volume is shared by the domU's.

This seems like a really powerful tool, and I'm wondering if anyone has tried it in their production virtualization environment, can think of any vendors (Citrix, VMWare) that take this approach, or can think of any serious problems with this idea. I can think of the following potential problems:

  • Potential filesystem problems if the "original" guest is running at snapshot time.
  • Host domain LVM performance as the "inverse delta" and "written block" logs are appended to. It may be very fast, but I don't know, and this seems like a weird idea, so it bears testing and benchmarking.
  • More eggs in one basket: if the "original" LVM volume gets corrupted, then the snapshotted volumes are hosed as well. Of course, RAID5 mitigates disk corruption, but it's worth noting.
  • Can you snapshot a snapshot? I assume so.

As people who likely have more virtualization experience than I, is there anything that makes want to run screaming from such a setup?

share|improve this question

3 Answers 3

up vote 3 down vote accepted

It's perfectly fine to do this. What you do not want is to have the parent of the snapshot (the original, or the source, or whatever you want to call it) to be in use at the same time, because it will cause IO multiplication (Hubert was right about this, it is just easy to prevent by not using the source volume all the time).

If you have one master OS install on an LVM, and you snapshot that four times, you will not have much of an IO penalty, since you are only writing to the individual snapshot volumes. Of course, it is not free, but neither are other forms of snapshotting on other filesystems or virtual disks. There is always a cost somewhere.

One other thing Hubert is right about, is that you have to think about the sizing of your snapshots. You will want to make sure the snapshot volumes are able to keep writing. A full snapshot volume will break stuff badly. A failsafe way to prevent this, is to make the snapshot volume the same size (or bigger) than the source volume. You loose the benefit of using less diskspace this way, though.

You know that qemu images are snapshot capable too?

share|improve this answer
    
Didn't know about qemu images, that's cool. Thanks for the detailed reply; good to know that LVM images can work. –  phred Oct 1 '10 at 17:19
1  
Also, other answerers, ZFS isn't the panacea you think it is. It's a great filesystem, but it's not and never will be a networked/clustered filesystem, and there are inherent synchronization problems with trying to share a Z filesystem between multiple VMs, just like ext3, just like reiser, etc. –  phred Oct 1 '10 at 17:36

While I didn't try LVM for KVM storage, I did use it for shadow volume feature of samba and I can tell you one thing: the performance was abyssal.

Each and every snapshot requires additional write to happen, if you have one base snapshoted volume and 4 snapshots, the amount of writes that goes to the drives is multiplied by 5 when you write to base volume.

As for specific questions:

  • LVM freezes a filesystem while the snapshot takes place (stops writes, flushes cache, does the snapshot, resumes writes)
  • as I said, it's very slow
  • Yes, corrupted base volume renders all snapshots unusable, what's more, if you run out of space allocated for the snapshot deltas, the snapshot is hosed as well
  • yes, you can snapshot a snapshot

Unfortunately, there are only 3 systems I know of that work well with snapshots: NetApp WAFL, ZFS and btrfs. If the system is not critical btrfs could be worth a shot.

share|improve this answer
1  
LVM snapshots work fine. I use them all the time. Your setup is just flawed. LVM snapshots are not meant for this. What works great for example, is to snapshot a MySQL data volume, back that up and then destroy the snapshot again. Having four snapshots around all of the time and then writing to the source: of course performance will be bad: you are quadrupling IO. –  wzzrd Sep 30 '10 at 9:33
    
Yes, LVM snapshots for backups are a good idea and work well (I use them all the time), but having them all the time trashes performance –  Hubert Kario Sep 30 '10 at 9:47
    
Yeah, LVM snapshots work amazingly well for MySQL for backups, agreed. Understood that the "source" of the snapshots needs to be quiescent to not totally kill performance, since yeah, it'd have to write the "negative blocks" back to all of the snapshotted volumes. ZFS snapshots wouldn't help in this circumstance, since ZFS "cannot provide concurrent access from multiple hosts" (according to Wikipedia), which is possible with LVM so long as the hosts all share a common hypervisor. –  phred Oct 1 '10 at 17:23
    
you can still create filesystem images on ZFS, this way ZFS doesn't differ in any way from raw LVM –  Hubert Kario Oct 1 '10 at 23:03

I would go so far as to say that ZFS could be worth a shot, its possible to setup a Nexenta based system backed by some kind of disk, either fiber, iscsi, etc, fairly easily, and has pretty good performance. I would recommend this approach over local storage if performance is not absolutely critical any day since this would give you a simple recovery scenario if your virtualization server went down.

share|improve this answer
1  
Except for the fact that you need an extra system... –  wzzrd Sep 30 '10 at 9:29
    
Yes, and once I have my Nexenta system (had to look it up: OpenSolaris kernel + Linux-y userland tools) I still have to export the filesystem to my Linux virtualization server, which means using NFS or CIFS, which has performance implications and won't allow me to do the operation I need, which is: stop VM from writing to the filesystem and snapshot the file on disk. Since ZFS isn't a network protocol, this means I need to script up some way to have the virtualization server run zfs snapshot on the fileserver, but that sounds like a nasty little house of cards. No thank you. –  phred Oct 1 '10 at 17:34
    
I don't consider having an extra system a bad thing, I guess it really depends on how many vms you are dealing with and what they are being used for. There are lots of ways to skin this cat. I generally think separating your storage from your server is a good thing in most cases even if its only at the fiber channel / iscsi level versus another layer of indirection that nexenta or some kind of managed filesystem platform would provide. Phred brings up very valid concerns, again implementation specific. For future reference there is a Nexenta API, it's relatively straight forward. –  MattyB Oct 3 '10 at 17:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.