I currently have one older server that provides VMs using Xen on CentOS. While not a monster, it supports VT extensions and has a 1TB hardware RAID1 disk set up. I am looking to add another server that is newer, keeping the old one for mutual backups and, potentially, for distributing the workload.
Thus far, suggestions have involved SANs or other kinds of costly hardware additions that I cannot afford. So, given the following hardware, operational goals, and constraints what is the best design? (one that minimizes cost and downtime and maximizes availability, performance, and stability)
Hardware
- Poweredge 850 1U server with 8GB RAM, CPU VT support, and 1TB RAID1
- Additional server
- not purchased yet, so this is flexible - think less than $3000
- considering a R410 with dual quad xeons, 16GB RAM and 4x1TB SATAs in RAID5 for 2.8TB
Operational requirements
- Servers must provide virtual machines
- Currently using Xen on CentOS 5
- Looked at Citrix XenServer, VMware Server and ESXi, KVM, headless VirtualBox
- The newer, more powerful server probably should be the 'main' one, hosting VMs that do all kinds of things including web and mail serving
- The purpose of getting a 2nd server is to gain some redundancy - if something happens to one, the other can take over for a while (think blown power supply and next day on-site warranty delay)
- When a VM is being backed up, it should either be continuously available or the downtime should be negligible (i.e. the time it takes to pause, start snapshot/clone/copy, unpause)
Constraints and considerations
- Am only interested in free solutions (open source preferred, but not strictly required)
- Space at the colo is billed per U, so adding 1U is preferred over larger servers. Larger hardware will only be considered if the solution is particularly slick.
- The number of VMs and the size of their disks makes transferring them off-site regularly over the internet infeasible due to bandwidth costs
- The two servers can be networked directly together so transfers between them are very fast and cost nothing
- The warranty on the older server is paid up for 2 years and it works fine, so let's not needlessly replace it (only really, really slick solutions that include replacing the old server would make sense for us)
- Not really considering a storage solution instead of a 2nd server because one server needs to be able to take over for the other if something should happen. If I only have one server and one storage solution, then I have 2 points of failure instead of 1.
Previous research
- The Xen version provided with CentOS (and on most dom0 supporting distros) is pretty old and crufty
- Current experience with Xen
- VM disks kept on logical volumes
- dd is slow and includes free space as well
- Mounting the filesystem in the dom0 and rsyncing requires the dom0 to know the domU's FS layout, and also gets really, really tricky if the domU is using LVM as well. Hard to automate, and doesn't necessarily result in a quickly reusable image on the 2nd server.
- The LVM snapshot -> backup -> remove snapshot process allows VMs to be available while doing incremental backups. Big plus!
- Citrix XenServer
- Makes it easier to pool resources, but requires shared storage and the processors to be basically the same. Unless I get another old server to match my current old server, I don't meet the requirements of XenMotion.
- Not sure if XenMotion really works for backups anyway. My understanding is that once the VM migrates, it has been moved, not copied, to the other server.
- Snapshot + export snapshot looks promising.
- Moving VMware VMs between running hypervisors requires paying for vmotion
- again, vmotion probably isn't intended for backups anyway
- KVM is the solution I know the least about, but seems to be very similar to Xen with respect to how it handles storage - local image files, logical volumes, or shared SAN/iSCSI
Phew! Thanks in advance for your comments! Let me know if you need more info :P
