KVM + LVM: Where to put LVM?

Question

Hy, I'm currently setting up a new Server using libvirt+kvm. Afterwards there should be about 5 virtual-machines on this server running (+some testing machines).

The storage is put on a raid-5 device which is set up using LVM. KVM now runs on some LVM-Logical Volumes.

The question is: Is there any drawback of using lvm again (a second time) inside the virtual machine to partition the space? So it would be: Harddisk-> Raid Controler -> LVM for Physical Server -> One Logical Volume per VM -> LVM inside each VM -> Several Logical Volumes inside each VM.

Are there any other possibilities if I want dynamic partitions inside my virtual machines?

Thanks

Answer 1

LVM's performance overhead is trivial, using it twice won't change that. Your raid-5 device is going to have a much much greater impact than lvm.

Answer 2

A blog article at http://hyperthese.net/post/kvmized-debian-on-lvm/ suggests that you create LVM logical volumes on the host (physical server) and create filesystems directly on them, without creating partitions, and before creating the virtual machines.

I tried it out, and long story short: seemed to work, and I was able to enlarge an LV and the filesystem on it.

Here's the long story, i.e. what I did:

Created LVM logical volumes for root, var and swap for the VM, on the host (running Ubuntu 10.04):

me@host:~$ sudo lvcreate -L4G -n test-root vg1
me@host:~$ sudo lvcreate -L20G -n test-var vg1
me@host:~$ sudo lvcreate -L2G -n test-swap vg1

Created file systems and swap on LVs:

me@host:~$ sudo mkfs.ext3 /dev/mapper/vg1-test--root
me@host:~$ sudo mkfs.ext3 /dev/mapper/vg1-test--var
me@host:~$ sudo mkswap -f /dev/mapper/vg1-test--swap

Created the VM:

me@host:~$ sudo virt-install --name=test --ram=2048 --os-type=linux --os-variant=ubuntulucid --cdrom=ubuntu-server-10.04-lts-64bit.iso --disk path=/dev/mapper/vg1-test--root --disk path=/dev/mapper/vg1-test--var --disk path=/dev/mapper/vg1-test--swap --network bridge=br0 --vnc --noautoconsole

Then I connected to the new VM using virt-viewer and the Ubuntu installer was there waiting. I chose the mode 'Install a minimal virtual machine' (F4 key).

At the partitioning phase I chose the manual partitioning. The installer found virtual disks vda, vdb and vdc and recognized the first two having ext3 and last as swap. I chose the ext3 partitions and told to use them as ext3 partitions (the default was "do not use"), with "no, keep the existing data" and mount point as / for the first one and /var for the second one. Swap was set up correctly by default. Then I chose to install grub on the first disk.

I got the VM up and running fine. Fdisk shows vda as having an empty partition table and vdb and vdc as not having a valid partition table. I don't know if having or not having a partitioning table is a problem, there's some discussion about it at https://unix.stackexchange.com/questions/5162/how-to-install-grub-to-a-whole-ext4-disk-without-partition-table .

Finally I tried resizing the var disk. First, on the host:

me@host:~$ sudo lvextend -L24G /dev/vg1/test-var

Then I rebooted the VM and resized the file system on the VM:

me@test:~$ sudo resize2fs /dev/vdb

And it resized it fine.

I don't know if this is a good way to do it or not, but so far it seems to work. Any comments?

Answer 3

I think, using a second time LVM won't be good for performance but except a network filesystem, I can't think of another stable solution with dynamic partition (you can try zfs with fuse or btrfs but there are not production ready).

If you want to keep LVM inside VM, you can create a LV on the host for each partition on each VM.

Answer 4

In general, using more parts mean more things to break. I'd advise rethinking what you're doing so that LVM isn't needed twice. Perhaps use something like OpenVZ instead of KVM, OpenVZ has support for resizing virtual partitions quickly and on the fly.