Clone kvm/qemu VM to a different server

Question

What's the easiest/quickest way to clone a VM from one server onto another if you don't have shared storage between the two servers (so you can't do a standard migration)?

I have a production ready VM installed on one server, and I want to clone it onto another system. I don't have shared storage between the two hosts, but I've copied the disk image between the two hosts and added a config for it (virsh defined it). When I try to start it however it doesn't take:

# virsh create /etc/libvirt/qemu/cloned-vm.xml 
error: Failed to create domain from /etc/libvirt/qemu/cloned-vm.xml
error: Unable to read from monitor: Connection reset by peer

I'm using KVM on RHEL6. Here is the duplicated config

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE 
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit 
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>cloned-vm</name>
  <uuid>NEW_UUID_HERE</uuid>
  <memory>15360000</memory>
  <currentMemory>15360000</currentMemory>
  <vcpu>7</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/local/vm/cloned-vm.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <interface type='bridge'>
      <mac address='NE:W_:MA:C_:AD:DR'/>
      <source bridge='br2'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Answer 1

And here is what I do where $VM is the VM name, and DEST is the destination hypervisor hostname. It does this on running VMs using dd and snapshot LVM disks (It is assuming that the LVM Group is called HypGroup00).

I just threw this together, so it isn't necessarily the prettiest, but does the job, I'm using it to migrate some VMs from a CentOS 5.9 hypervisor to CentOS 6 with minimal downtime.

This is for CentOS 5.9, tested on going to CentOS 5 and 6 as a destination.

VM=webserver
DEST=hyp5

Figure out which disks to copy, since most of our VM disks are referenced via the /dev/mapper/ path rather than the /dev/volgroup/volname path. So we need to translate between the two.

DISKS=`cat /etc/libvirt/qemu/$VM.xml |  grep HypGroup00 | sed "s|.*/\(HypGroup00-.*\)'/>|\1|"`

LVS=""

for disk in $DISKS; do
  echo VM Disk $disk
  LV=`lvdisplay /dev/mapper/$disk | grep "LV Name" | awk '{print $3}'`
  LVS="$LVS $LV"
done

Copy over the VM definition and register it

virsh dumpxml $VM > /tmp/$VM.xml
scp /tmp/$VM.xml $DEST:/tmp/
ssh $DEST virsh undefine $VM > /dev/null 2>&1
ssh $DEST virsh define /tmp/$VM.xml

Now create the LV on the remote server

for lv in $LVS; do
  ssh $DEST lvremove --force $lv
  SIZE=`lvdisplay $lv | grep "LV Size" | awk '{print $3}'`
  SHORTNAME=`basename $lv`
  ssh $DEST lvcreate -L"$SIZE"G -n$SHORTNAME HypGroup00
done

Now create the snapshot LVs and start copying over the data

virsh suspend $VM
for lv in $LVS; do
  lvcreate -L10G -s -n $lv-snapshot $lv
done
virsh resume $VM

Copy the disks across

for lv in $LVS; do
  echo Copying LV $lv, this will take a while...
  time dd bs=1M if=$lv-snapshot | gzip --fast | ssh $DEST "gzip -d | dd of=$lv"
done

A more complex version of the above that shows progress if required from dd, no good for multiple copies because of /tmp/pid, but you can change to to include $$ if you wish.

(dd bs=1M if=$lv-snapshot & echo $! >&3 ) 3>/tmp/pid  2> >(grep 'copied' 1>&2) | ssh $DEST "dd bs=1M of=$lv" &
# Need this sleep to give the above time to run
sleep 1
PID=$(</tmp/pid)

while kill -0 $PID; do
  kill -USR1 $PID
  sleep 5
done

Clean up

for lv in $LVS; do
  lvremove --force $lv-snapshot
done

Answer 2

Okay so the way I was doing it actually did work just fine. The problem is just that I didn't have enough resources to run that VM. So just to answer my own question... here are the details to how I did the VM duplication across different servers without shared disk.

Because you don't have a shared disk, you can't do a typical 'clone' and then 'migration'. Instead you do a typical clone

1. Here's the command to do the cloning (/local/vm/ is the path to your VM images, usually /var/something/):

   virt-clone --original=vm-to-clone --name=cloned-vm -f /local/vm/cloned-vm.img --mac=xx:xx:xx:xx:xx:xx

2. Now copy that img file from one server to the other... my servers can't talk directly to eachother, so I use this little SSH redirect to do the trick:

   ssh -n server1 '(cd /local/vm/; cat cloned-vm.img)' | ssh server2 '(cd /local/vm/; cat > cloned-vm.img)'

3. Then copy the config for that VM over:

   ssh -n server1 '(cd /etc/libvirt/qemu/; cat cloned-vm.xml)' | ssh server2 '(cd /etc/libvirt/qemu/; cat > cloned-vm.xml)'

4. Update the config with any new changes. In my case (and this was what was causing my problem), I needed to lower the "memory" and "currentMemory" attributes.

5. Add the new VM to libvirt:

   virsh define /etc/libvirt/qemu/cloned-vm.xml

6. Run it:

   virsh create /etc/libvirt/qemu/cloned-vm.xml