Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm planning on setting up a XenServer machine that uses a NexentaOS/ZFS-based SAN for storage of the Virtual Disk Images (VDIs) through iSCSI. I know I could just setup a big Storage Repository (SR) on the SAN and let XenServer take care of snapshotting and cloning disk images. However, I'd love to tap more into the power of ZFS and use that for snapshotting/cloning, for a few reasons:

  1. I'm not sure how XenServer's snapshotting/cloning works, but if it's based on LVM, I'm concerned I'd run into issues when dealing with multiple snapshots. I've done some experiments a while ago with multiple LVM snapshots of the same data and performance seemed poor and the snapshots wasted a lot of space. It seems that ZFS snapshots are far superior to LVM snapshots.
  2. The SAN would be taking automatic (and efficient) periodic ZFS snapshots that could go back in time a while, and I'd love to be able to revert a VM to such ZFS snapshot.

Would letting ZFS handle snapshotting/cloning instead of doing it through XenServer be advisable, and if so, what's the best way to go about it? If I put all VDIs inside a single large SR, and take ZFS snapshots of the entire SR, I would not be able to roll back an individual VM at a time. I could create one SR per VDI, and if I have to rollback a VDI, I'd carefully detach the SR, roll it back on the SAN, and re-attach it. However, I'm guessing that I'd run into problems when attaching a cloned SR if XenServer detects duplicate SR UUIDs. Are there any better ways to handle cloning or rolling back to previous snapshots from the SAN?

share|improve this question

4 Answers 4

up vote 3 down vote accepted

As other answers alluded to, the ideal approach is LUN-per-VDI. At first it didn't look like it was possible to do this, but there is an undocumented "iscsi" SR driver that will create a LUN-per-VDI SR (I found this when looking through the /opt/xensource/sm directory - see the ISCSISR.py file). You essentially setup one SR for each iSCSI target, and XenServer creates the VDIs for each LUN on that target. You can only set this up through the command-line, including creating the VBDs and attaching it to VMs. The VBDs and VDIs don't even show up in XenCenter.

Here's a sample command to set it up:

xe sr-create name-label=san type=iscsi device-config:target=<IP address of target> device-config:targetIQN=<IQN of target> shared=true

This will automatically create an VDI for each iSCSI LUN. If you end up adding a new iSCSI LUN on the SAN, XenServer will add a new VDI for it after executing the following command:

xe sr-scan uuid=<UUID of SR>

This also works when adding VDIs for cloned LUNs - the new VDI gets assigned a new UUID.

Also, if you end up resizing a LUN, XenServer does not automatically pick up on that, so you'd have to execute the following:

xe vdi-forget uuid=<UUID of VDI that you resized>
iscsiadm -m node -R
xe sr-scan uuid=<UUID of SR>

And to create a VBD and attach it to a VM:

xe vbd-create vm-uuid=<UUID of VM> device=<VBD device #, e.g. 0> vdi-uuid=<VDI of UUID>
share|improve this answer

I've done several configuration with xenserver similar to this setup

And I've used one of two methods

  1. If I dont have many vm's I create the system disk with VDI per VM and data disk as direct attach Iscsi lun

  2. If I have lots of vm (20+) I create a big SR and when I need to roll back I can

    A. rename the VG before connecting to xenserver (vgrename on a different machine - even virtual )

    B. You can even attach the big SR snapshot to a virtual machine and export it again with iscsi ,

I hope its not too complicated :)

share|improve this answer
Thanks, that's helpful. For option #1, how did you setup the direct attach iSCSI LUNs as VDI (i.e. which xe commands did you use)? And for option #2A, does XenServer essentially store the UUID in the VG name, so this works for cloning because when you rename the VG, you essentially change the UUID? –  Alex May 21 '11 at 14:12
for option 1 the direct attach Iscsi , I use the virtual machine Iscsi initiator ( Linux/windows iscsi initiator ) - for option 2A yes, look at your vg names you will see it matches your sr-uuid's –  John May 21 '11 at 15:58
Interesting, so are you saying for option 1, you essentially have no VBD for the VM (i.e. you have a diskless VM as far as XenServer is concerned, other than maybe a small boot drive), and instead run the iSCSI initiator from within the VM? Have you tested to see how performance compares between options 1 and 2? –  Alex May 22 '11 at 6:07

Interesting configuration.. specially for ZFS.

According to my experience with XenServer, the best you can do is let the storage system manage the disk (and it snapshots and other administrative tasks).

I think you should let ZFS handle te snapshots and cloning, but I'm not sure if the best you can do is give to XenServer one big SR (I agree with what you said about roll back). This is a very complex setup, because if you have a lot of VM you'll have a lot of VDI, the administration could be a mess but you will earn the capability to rollback. I'd use one VDI per VM.

Answering your question, if you double check that you detach the SR when rolling back you wont have any problems when attaching a cloned SR. I've done that before, not in XenServer but in XCP (http://goo.gl/4wfE)

share|improve this answer
Thanks for the input. With regards to cloning, the problem I anticipate is that a single SR would turn into two SRs, and I wouldn't detach the original SR because I'd want to duplicate the VDI for a second VM. With snapshotting, I'd detach the original SR, but with cloning, I wouldn't, hence the potential UUID conflict I mentioned. Any way around that? –  Alex May 21 '11 at 5:04

Our customers (Nexenta customers) who do not have very many VMs, fewer than 1000 mostly, choose to do a LUN per VM at times. This allows you to snapshot easily from NexentaStor of each individual VM. What's cool is that you can create one golden image, aka clome-master. Then use that clone-master to spin up new VMs, by simply cloning it. And the added benefit is that you are not using any additional disk space.

share|improve this answer
Are you talking about one SR per LUN, and only one VM per SR? If yes, wouldn't you run into the UUID conflict problem I mentioned in my question (and for which a workaround was presented in one of the answers)? If no, would you please explain how customers attach a LUN directly to a VM in XenServer without using a SR? –  Alex May 22 '11 at 23:29
Yes, one SR per LUN, and one VM per SR. I have not encountered UUID issues that you describe. Though, most customers are ESX users, so quite possibly I was not exposed enough to the Xenserver customers to speak intelligently here. –  slashdot May 24 '11 at 12:54

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.