I currently have one older server that provides VMs using Xen on CentOS. While not a monster, it supports VT extensions and has a 1TB hardware RAID1 disk set up. I am looking to add another server that is newer, keeping the old one for mutual backups and, potentially, for distributing the workload.

Thus far, suggestions have involved SANs or other kinds of costly hardware additions that I cannot afford. So, given the following hardware, operational goals, and constraints what is the best design? (one that minimizes cost and downtime and maximizes availability, performance, and stability)


  • Poweredge 850 1U server with 8GB RAM, CPU VT support, and 1TB RAID1
  • Additional server
    • not purchased yet, so this is flexible - think less than $3000
    • considering a R410 with dual quad xeons, 16GB RAM and 4x1TB SATAs in RAID5 for 2.8TB

Operational requirements

  • Servers must provide virtual machines
    • Currently using Xen on CentOS 5
    • Looked at Citrix XenServer, VMware Server and ESXi, KVM, headless VirtualBox
  • The newer, more powerful server probably should be the 'main' one, hosting VMs that do all kinds of things including web and mail serving
  • The purpose of getting a 2nd server is to gain some redundancy - if something happens to one, the other can take over for a while (think blown power supply and next day on-site warranty delay)
  • When a VM is being backed up, it should either be continuously available or the downtime should be negligible (i.e. the time it takes to pause, start snapshot/clone/copy, unpause)

Constraints and considerations

  • Am only interested in free solutions (open source preferred, but not strictly required)
  • Space at the colo is billed per U, so adding 1U is preferred over larger servers. Larger hardware will only be considered if the solution is particularly slick.
  • The number of VMs and the size of their disks makes transferring them off-site regularly over the internet infeasible due to bandwidth costs
  • The two servers can be networked directly together so transfers between them are very fast and cost nothing
  • The warranty on the older server is paid up for 2 years and it works fine, so let's not needlessly replace it (only really, really slick solutions that include replacing the old server would make sense for us)
  • Not really considering a storage solution instead of a 2nd server because one server needs to be able to take over for the other if something should happen. If I only have one server and one storage solution, then I have 2 points of failure instead of 1.

Previous research

  • The Xen version provided with CentOS (and on most dom0 supporting distros) is pretty old and crufty
  • Current experience with Xen
    • VM disks kept on logical volumes
    • dd is slow and includes free space as well
    • Mounting the filesystem in the dom0 and rsyncing requires the dom0 to know the domU's FS layout, and also gets really, really tricky if the domU is using LVM as well. Hard to automate, and doesn't necessarily result in a quickly reusable image on the 2nd server.
    • The LVM snapshot -> backup -> remove snapshot process allows VMs to be available while doing incremental backups. Big plus!
  • Citrix XenServer
    • Makes it easier to pool resources, but requires shared storage and the processors to be basically the same. Unless I get another old server to match my current old server, I don't meet the requirements of XenMotion.
    • Not sure if XenMotion really works for backups anyway. My understanding is that once the VM migrates, it has been moved, not copied, to the other server.
    • Snapshot + export snapshot looks promising.
  • Moving VMware VMs between running hypervisors requires paying for vmotion
    • again, vmotion probably isn't intended for backups anyway
  • KVM is the solution I know the least about, but seems to be very similar to Xen with respect to how it handles storage - local image files, logical volumes, or shared SAN/iSCSI

Phew! Thanks in advance for your comments! Let me know if you need more info :P

  • Colo is a suburb of Sydney, Australia.
    – pavium
    Nov 19, 2009 at 2:50
  • I think you're thinking of Como.
    – womble
    Nov 19, 2009 at 7:58

2 Answers 2


You could setup DRBD between the two servers to host the VM images and configuration files.

I believe this setup will allow for live migration between the two hosts. If not it should allow you to just start a VM up on either server should one go down. This could be automated a bit by using heartbeat to run some scripts to restart the VM's should one of the hosts go down. This article seems to do this with live migration and LVM.

  • DRBD is awesome, we use it extensively. I'd recommend against sharing the Xen config files, though; that hasn't worked well for us (just enough chance of them needing to be different to screw you over). Also, leave the swap LVs local (if you're using HVM, give it a separate virtual disk) to avoid having DRBD getting hammered with swap IO that will never survive a VM restart anyway -- unless you're dead-set on going with live migration -- which on a two-node cluster, I don't see the value in.
    – womble
    Nov 19, 2009 at 7:57
  • You can only do redundancy with DRBD if you use a clustering filesystem such as OCFS or GFS. With ext2/3, all you will have is a redundancy in case the whole server goes down. Nov 19, 2009 at 8:15
  • Thanks for the quick answer, 3dinfluence. I previously hadn't seen DRBD and I think it is just what we have been looking for. We now just have to make sure that the old backup server is fast enough to keep up with the proposed new server and I think we'll be all set. Also, found this email ( archives.free.net.ph/message/20080820.180702.da874e6f.fi.html ) which reassures me that we shouldn't have any processor similarity issues with DRBD.
    – user26585
    Nov 20, 2009 at 22:00

We have a two-node cluster built about a year ago with CentOS 5.2, Xen 3.2, LVM and DRBD 8.2.6... I used this howto as a guide for setting everything up, even though the guide itself is for Ubuntu Hardy, but CentOS' Xen support is far more stable IMO.

  • Thanks for the link - bookmarked it. How are you getting Xen 3.2? Building it yourself or using a repository other than the main one?
    – user26585
    Nov 20, 2009 at 21:59
  • I used the RPMs from gitco.de/repo/xen3.2.1
    – Joril
    Nov 21, 2009 at 11:06

You must log in to answer this question.