# How to manage Xen virtual machines on shared SAN storage?

We have a small cluster of Xen servers in the office. Each virtual machine system image is on its own block device on our iSCSI SAN. The servers all connect their iSCSI initiator to every VM LUN simultaneously.

To prevent the same VM from being started on multiple hosts we just ensure that each virtual machine is only added to one server through the use of xm new. Judicious monitoring of all the servers through virt-manager helps as well. However, there really there are no safeguards that would prevent someone from accidentally launching the same VM on multiple hosts.

My question is, what would be a good way to implement a better management system for a setup like this? Are there any existing tools that will manage a Xen setup like this with a shared storage back-end?

We're using openSUSE 11.1 as the host OS.

## 2 Answers

Xen has a built-in locking mechanism that should be used, especially if using hot or live migration. We have a NFS share mounted on all of our Dom0 hosts, and Xen checks there first to see if someone else is running an instance of a DomU before attempting to start it.

Look in your xend-config.sxp file -- the config settings your looking for start at about line 261 on my system, and are called xend-domain-lock, xend-domain-lock-path, and if you want to set up an external utility to manage it in another way, you can specify an external command to run with xend-domain-lock-utility. I'm using Xen 3.3.1 on SLES11; your mileage may vary.

We manage xend, xendomains, and nfs with pacemaker/openais, and store the lock files on an NFS mount that is made highly available by means of pacemaker and a partition on the san.

• Where is this built in locking mechanism? I've never seen this mentioned in any documentation, and I've been told that such a thing is not built in to Xen. Do you have some references? – Kamil Kisiel Jun 8 '09 at 16:43
• Look in your xend-config.sxp file -- the config settings your looking for start at about line 261 on my system, and are called xend-domain-lock, xend-domain-lock-path, and if you want to set up an external utility to manage it in another way, you can specify an external command to run with xend-domain-lock-utility. – Karl Katzke Jun 8 '09 at 17:20
• ... I'm using Xen 3.3.1, YMMV with earlier versions. – Karl Katzke Jun 8 '09 at 17:21
• Perfect! That's exactly what I was looking for. The xend-domain-lock-utility could presumably even hook in to a clustered volume manager or something similar. I've accepted the answer, but maybe you can update it with the specific configuration bits out of the comments just to make it a bit more clear? – Kamil Kisiel Jun 9 '09 at 4:47
• I went ahead and updated things. Let me know if you need any more details. – Karl Katzke Jun 9 '09 at 13:08

You have quite a few management tools for Xen clusters :

Convirt is nice for smaller setups. OpenQRM and Enomalism are for bigger setups.