Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm planning on setting up a XenServer machine that uses a NexentaOS/ZFS-based SAN for storage of the Virtual Disk Images (VDIs) through iSCSI. I know I could just setup a big Storage Repository (SR) on the SAN and let XenServer take care of snapshotting and cloning disk images. However, I'd love to tap more into the power of ZFS and use that for snapshotting/cloning, for a few reasons:

  1. I'm not sure how XenServer's snapshotting/cloning works, but if it's based on LVM, I'm concerned I'd run into issues when dealing with multiple snapshots. I've done some experiments a while ago with multiple LVM snapshots of the same data and performance seemed poor and the snapshots wasted a lot of space. It seems that ZFS snapshots are far superior to LVM snapshots.
  2. The SAN would be taking automatic (and efficient) periodic ZFS snapshots that could go back in time a while, and I'd love to be able to revert a VM to such ZFS snapshot.

Would letting ZFS handle snapshotting/cloning instead of doing it through XenServer be advisable, and if so, what's the best way to go about it? If I put all VDIs inside a single large SR, and take ZFS snapshots of the entire SR, I would not be able to roll back an individual VM at a time. I could create one SR per VDI, and if I have to rollback a VDI, I'd carefully detach the SR, roll it back on the SAN, and re-attach it. However, I'm guessing that I'd run into problems when attaching a cloned SR if XenServer detects duplicate SR UUIDs. Are there any better ways to handle cloning or rolling back to previous snapshots from the SAN?

share|improve this question
    
See this link for how to rename the UUIDs of a downlevel SAN cloned SR so that it can be attached and used at the same time as the uplevel SR is attached to and being used by the same XenServer: discussions.citrix.com/topic/… –  pekowski Jun 2 at 4:31

5 Answers 5

up vote 3 down vote accepted

As other answers alluded to, the ideal approach is LUN-per-VDI. At first it didn't look like it was possible to do this, but there is an undocumented "iscsi" SR driver that will create a LUN-per-VDI SR (I found this when looking through the /opt/xensource/sm directory - see the ISCSISR.py file). You essentially setup one SR for each iSCSI target, and XenServer creates the VDIs for each LUN on that target. You can only set this up through the command-line, including creating the VBDs and attaching it to VMs. The VBDs and VDIs don't even show up in XenCenter.

Here's a sample command to set it up:

xe sr-create name-label=san type=iscsi device-config:target=<IP address of target> device-config:targetIQN=<IQN of target> shared=true

This will automatically create an VDI for each iSCSI LUN. If you end up adding a new iSCSI LUN on the SAN, XenServer will add a new VDI for it after executing the following command:

xe sr-scan uuid=<UUID of SR>

This also works when adding VDIs for cloned LUNs - the new VDI gets assigned a new UUID.

Also, if you end up resizing a LUN, XenServer does not automatically pick up on that, so you'd have to execute the following:

xe vdi-forget uuid=<UUID of VDI that you resized>
iscsiadm -m node -R
xe sr-scan uuid=<UUID of SR>

And to create a VBD and attach it to a VM:

xe vbd-create vm-uuid=<UUID of VM> device=<VBD device #, e.g. 0> vdi-uuid=<VDI of UUID>
share|improve this answer

I've done several configuration with xenserver similar to this setup

And I've used one of two methods

  1. If I dont have many vm's I create the system disk with VDI per VM and data disk as direct attach Iscsi lun

  2. If I have lots of vm (20+) I create a big SR and when I need to roll back I can

    A. rename the VG before connecting to xenserver (vgrename on a different machine - even virtual )

    B. You can even attach the big SR snapshot to a virtual machine and export it again with iscsi ,

I hope its not too complicated :)

share|improve this answer
    
Thanks, that's helpful. For option #1, how did you setup the direct attach iSCSI LUNs as VDI (i.e. which xe commands did you use)? And for option #2A, does XenServer essentially store the UUID in the VG name, so this works for cloning because when you rename the VG, you essentially change the UUID? –  Alex May 21 '11 at 14:12
    
for option 1 the direct attach Iscsi , I use the virtual machine Iscsi initiator ( Linux/windows iscsi initiator ) - for option 2A yes, look at your vg names you will see it matches your sr-uuid's –  John May 21 '11 at 15:58
    
Interesting, so are you saying for option 1, you essentially have no VBD for the VM (i.e. you have a diskless VM as far as XenServer is concerned, other than maybe a small boot drive), and instead run the iSCSI initiator from within the VM? Have you tested to see how performance compares between options 1 and 2? –  Alex May 22 '11 at 6:07

Interesting configuration.. specially for ZFS.

According to my experience with XenServer, the best you can do is let the storage system manage the disk (and it snapshots and other administrative tasks).

I think you should let ZFS handle te snapshots and cloning, but I'm not sure if the best you can do is give to XenServer one big SR (I agree with what you said about roll back). This is a very complex setup, because if you have a lot of VM you'll have a lot of VDI, the administration could be a mess but you will earn the capability to rollback. I'd use one VDI per VM.

Answering your question, if you double check that you detach the SR when rolling back you wont have any problems when attaching a cloned SR. I've done that before, not in XenServer but in XCP (http://goo.gl/4wfE)

share|improve this answer
    
Thanks for the input. With regards to cloning, the problem I anticipate is that a single SR would turn into two SRs, and I wouldn't detach the original SR because I'd want to duplicate the VDI for a second VM. With snapshotting, I'd detach the original SR, but with cloning, I wouldn't, hence the potential UUID conflict I mentioned. Any way around that? –  Alex May 21 '11 at 5:04

Our customers (Nexenta customers) who do not have very many VMs, fewer than 1000 mostly, choose to do a LUN per VM at times. This allows you to snapshot easily from NexentaStor of each individual VM. What's cool is that you can create one golden image, aka clome-master. Then use that clone-master to spin up new VMs, by simply cloning it. And the added benefit is that you are not using any additional disk space.

share|improve this answer
    
Are you talking about one SR per LUN, and only one VM per SR? If yes, wouldn't you run into the UUID conflict problem I mentioned in my question (and for which a workaround was presented in one of the answers)? If no, would you please explain how customers attach a LUN directly to a VM in XenServer without using a SR? –  Alex May 22 '11 at 23:29
    
Yes, one SR per LUN, and one VM per SR. I have not encountered UUID issues that you describe. Though, most customers are ESX users, so quite possibly I was not exposed enough to the Xenserver customers to speak intelligently here. –  slashdot May 24 '11 at 12:54

This seems relevant. The bash script below has functions for attaching and detaching a SR as well as renaming all the UUIDs in an existing (for example SAN cloned) SR. It could be used to rename the UUIDs of an old clone and attach it without conflicting with the latest version of the volume being attached.

    # returns scsi_id, scsi_lun_id and as a bonus host_uuid
    probe_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4

            cmd="ssh root@$xen_host_ip xe sr-probe type=lvmoiscsi device-config:target=$iscsi_device_ip device-config:targetIQN=$iscsi_device_target_iqn:$iscsi_vol_name"
            echo $cmd
            probe_out=`$cmd 2>&1`
            #echo probe_out=$probe_out

            scsi_id=`echo $probe_out | awk '{for(i=1;i<=NF;i++){if($i=="<SCSIid>"){i+=1;o=o","$i}}}END{print substr(o,2)}'`
            echo scsi_id=$scsi_id
            if [ -z $scsi_id ];then
                    exit
            fi
            # Only allow one LUN per target. To allow for more, enhance input to specify volume serial number and search probe output for it.
            if [ `echo $scsi_id | awk -F, '{print NF}'` != 1 ];then
                    echo Only one LUN per iscsi target is supported
                    exit
            fi

            scsi_lun_id=`echo $probe_out | awk '{for(i=1;i<=NF;i++){if($i=="<LUNid>"){i+=1;o=o" "$i}}}END{print substr(o,2)}'`
            echo scsi_lun_id=$scsi_lun_id
            if [ -z $scsi_lun_id ];then
                    exit
            fi

            # This only works if it is a standalone XenServer, not in a Xen pool - so this becomes a requirement
            host_uuid=`ssh root@$xen_host_ip xe host-list --minimal`
            if [ -z $host_uuid ];then
                    echo host_uuid=$host_uuid
                    exit
            fi
    }


    # Create SR wipes out contents of previous SR
    create_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            probe_sr $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name

            cmd="ssh root@$xen_host_ip xe sr-create host-uuid=$host_uuid content-type=user name-label=$sr_name shared=true device-config:target=$iscsi_device_ip device-config:targetIQN=$iscsi_device_target_iqn:$iscsi_vol_name device-config:SCSIid=$scsi_id type=lvmoiscsi"
            echo $cmd
            $cmd
    }

    # mainly returns sr_uuid, but also pv_device used in uuid_regen_sr
    get_sr_uuid_from_file_system(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            scsi_lun_id=$5

            # Ensure the the VG for the new volume gets created
            ##iscsiadm -m discovery -t sendtargets -p $iscsi_device_ip
            echo ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --login
            ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --login

            # get the pv_device name
            # Some iSCSI SAN might need customization here
            #scsi_id_short=`echo $scsi_id | awk '{print substr($1,1,length("36589cfc000000"))}'`
            scsi_id_short=$scsi_id
            echo scsi_id_short=$scsi_id_short
            unset pv_device_short
            tries=1
            while [ $tries -le 3 ]&&[ -z $pv_device_short ];do
                    echo "ssh root@$xen_host_ip ls -l /dev/disk/by-scsibus/ | egrep \"$scsi_id_short.*-[0-9]+:[0-9]+:[0-9]+:$scsi_lun_id -> \" | awk '{print \$NF}' | awk -F/ '{print \$NF}'"
                    pv_device_short=`ssh root@$xen_host_ip ls -l /dev/disk/by-scsibus/ | egrep "$scsi_id_short.*-[0-9]+:[0-9]+:[0-9]+:$scsi_lun_id -> " | awk '{print $NF}' | awk -F/ '{print $NF}'`
                    pv_device=/dev/$pv_device_short
                    echo pv_device=$pv_device
                    if [ -z "$pv_device_short" ];then
                            sleep 1
                    fi
                    tries=$(( tries + 1 ))
            done
            if [ -z "$pv_device_short" ];then
                    exit
            fi
            if [ `echo $pv_device | wc -l` -gt 1 ];then
                    exit
            fi

            # pvscan ensure PVs, VGs and LVMs are visible
            echo ssh root@$xen_host_ip pvscan
            ssh root@$xen_host_ip pvscan 2>/dev/null

            # get the VG UUID which is also what Xen uses for the SR UUID
            sr_uuid=`ssh root@$xen_host_ip "pvs 2>/dev/null" | egrep "$pv_device.*VG_XenStorage-" | awk '{print substr($2,length("VG_XenStorage-")+1)}'`
            echo sr_uuid=$sr_uuid
            if [ -z "$sr_uuid" ];then
                    exit
            fi

            # Thought this might be good, but think it caused problems.  Better and seemingly required place is in detach_sr
            #echo ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --logout
            #ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --logout
    }

    attach_sr_xen(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            # Attach volume as a Xen SR
            echo ssh root@$xen_host_ip "xe sr-introduce uuid=$sr_uuid name-label=\"$sr_name\" type=lvmoiscsi shared=true"
            ssh root@$xen_host_ip "xe sr-introduce uuid=$sr_uuid name-label=\"$sr_name\" type=lvmoiscsi shared=true"

            echo ssh root@$xen_host_ip "xe pbd-create sr-uuid=$sr_uuid host-uuid=$host_uuid type=lvmoiscsi device-config-target=$iscsi_device_ip device-config-targetIQN=$iscsi_device_target_iqn:$iscsi_vol_name device-config-SCSIid=$scsi_id"
            pbd_uuid=`ssh root@$xen_host_ip "xe pbd-create sr-uuid=$sr_uuid host-uuid=$host_uuid type=lvmoiscsi device-config-target=$iscsi_device_ip device-config-targetIQN=$iscsi_device_target_iqn:$iscsi_vol_name device-config-SCSIid=$scsi_id"`
            echo pbd_uuid=$pbd_uuid
            if [ -z $pbd_uuid ];then
                    exit
            fi

            echo ssh root@$xen_host_ip "xe pbd-plug uuid=$pbd_uuid"
            ssh root@$xen_host_ip "xe pbd-plug uuid=$pbd_uuid"
    }

    # Attach SR retains prior contents as opposed to create_sr
    attach_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            probe_sr $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name
            get_sr_uuid_from_file_system $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $scsi_lun_id
            attach_sr_xen $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $sr_name
    }

    # Detach/forget SR
    detach_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            echo ssh root@$xen_host_ip "xe sr-list name-label=\"$sr_name\" --minimal"
            sr_uuid=`ssh root@$xen_host_ip "xe sr-list name-label=\"$sr_name\" --minimal"`
            echo sr_uuid=$sr_uuid
            if [ -z $sr_uuid  ];then
                    exit
            fi
            echo ssh root@$xen_host_ip "xe pbd-list sr-uuid=$sr_uuid --minimal"
            pbd_uuid=`ssh root@$xen_host_ip "xe pbd-list sr-uuid=$sr_uuid --minimal"`
            echo pbd_uuid=$pbd_uuid
            if [ -z $pbd_uuid ];then
                    echo No PBD found, proceeding anyway
            fi
            echo ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --logout
            ssh root@$xen_host_ip iscsiadm -m node -T $iscsi_device_target_iqn:$iscsi_vol_name -p $iscsi_device_ip:3260 --logout
            if [ ! -z $pbd_uuid ];then
                    echo ssh root@$xen_host_ip "xe pbd-unplug uuid=$pbd_uuid"
                    ssh root@$xen_host_ip "xe pbd-unplug uuid=$pbd_uuid"
            fi
            echo ssh root@$xen_host_ip "xe sr-forget uuid=$sr_uuid"
            ssh root@$xen_host_ip "xe sr-forget uuid=$sr_uuid"

            echo "It is now ok to remove the volume from the storage array"

    }

    # generates new PV, SR (VG) and VDI (LVM) uuids for a cloned and unattached iSCSI SR, leaving it unattached
    uuid_regen_sr(){
            xen_host_ip=$1
            iscsi_device_ip=$2
            iscsi_device_target_iqn=$3
            iscsi_vol_name=$4
            sr_name=$5

            probe_sr $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name

            get_sr_uuid_from_file_system $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $scsi_lun_id

            # Import the clone giving it a new PV ID and VG UUID
            new_sr_uuid=`uuidgen`
            echo ssh root@$xen_host_ip vgimportclone --basevgname VG_XenStorage-$new_sr_uuid --import $pv_device
            ssh root@$xen_host_ip vgimportclone --basevgname VG_XenStorage-$new_sr_uuid --import $pv_device

            # get the new VG UUID which is also what Xen uses for the SR UUID
            sr_uuid=`ssh root@$xen_host_ip "pvs 2>/dev/null" | egrep "$pv_device.*VG_XenStorage-" | awk '{print substr($2,length("VG_XenStorage-")+1)}'`
            echo sr_uuid=$sr_uuid
            if [ -z $sr_uuid ]||[ $sr_uuid != $new_sr_uuid ];then
                    echo bad sr_uuid
                    exit
            fi

            attach_sr_xen $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $sr_name

            # pvscan to get rid of some Linux duplicate dev errors
            echo ssh root@$xen_host_ip pvscan
            ssh root@$xen_host_ip pvscan 2>/dev/null

            echo -n waiting for VDI count to settle :
            prev_num_vdis=`ssh root@$xen_host_ip "xe vdi-list sr-uuid=$sr_uuid" | grep name-label | wc -l`
            echo -n $prev_num_vdis" "
            sleep 5
            num_vdis=`ssh root@$xen_host_ip "xe vdi-list sr-uuid=$sr_uuid" | grep name-label | wc -l`
            echo -n $num_vdis" "
            while [ $num_vdis != $prev_num_vdis ];do
                    sleep 5
                    num_vdis=`ssh root@$xen_host_ip "xe vdi-list sr-uuid=$sr_uuid" | grep name-label | wc -l`
                    echo -n $num_vdis" "
                    prev_num_vdis=$num_vdis
            done
            echo ""

            metadata_sr_uuid=`echo $sr_uuid | awk '{for(i=1;i<=length($1);i++){c=substr($1,i,1);if(c=="-"){o=o"--"}else{o=o""c}};print o}'`
            echo ssh root@$xen_host_ip "cp VG_XenStorage--$metadata_sr_uuid""-MGT ."
            ssh root@$xen_host_ip "cp /dev/mapper/VG_XenStorage--$metadata_sr_uuid""-MGT ."

            for vdi_uuid in `ssh root@$xen_host_ip lvs VG_XenStorage-$sr_uuid | grep VHD\- | awk '{print substr($1,length("VHD-")+1)}'`;do
                    new_uuid=`uuidgen`
                    echo ssh root@$xen_host_ip "sed -i \"s/$vdi_uuid/$new_uuid/g\" VG_XenStorage--$metadata_sr_uuid""-MGT"
                    ssh root@$xen_host_ip "sed -i \"s/$vdi_uuid/$new_uuid/g\" VG_XenStorage--$metadata_sr_uuid""-MGT"

                    echo ssh root@$xen_host_ip "lvrename /dev/VG_XenStorage-$sr_uuid/VHD-$vdi_uuid /dev/VG_XenStorage-$sr_uuid/VHD-$new_uuid"
                    ssh root@$xen_host_ip "lvrename /dev/VG_XenStorage-$sr_uuid/VHD-$vdi_uuid /dev/VG_XenStorage-$sr_uuid/VHD-$new_uuid"
            done

            echo ssh root@$xen_host_ip "cp VG_XenStorage--$metadata_sr_uuid""-MGT /dev/mapper/VG_XenStorage--$metadata_sr_uuid""-MGT"
            ssh root@$xen_host_ip "cp VG_XenStorage--$metadata_sr_uuid""-MGT /dev/mapper/VG_XenStorage--$metadata_sr_uuid""-MGT"

            echo ssh root@$xen_host_ip "rm -f VG_XenStorage--$metadata_sr_uuid""-MGT"
            ssh root@$xen_host_ip "rm -f VG_XenStorage--$metadata_sr_uuid""-MGT"

            echo ssh root@$xen_host_ip "xe sr-scan uuid=$sr_uuid"
            ssh root@$xen_host_ip "xe sr-scan uuid=$sr_uuid"

            detach_sr $xen_host_ip $iscsi_device_ip $iscsi_device_target_iqn $iscsi_vol_name $sr_name
}
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.