Understanding the concept: Bare-metal VMware ESXi 5.0 with FreeNAS guest VM and iSCSI

Question

I have a test setup (HP Microserver) with just a single disk at the moment and have ESXi 5.0 bare-metal installed on a USB flash drive and a FreeNAS 8 VM created as a 2GB install, but now I'm at a loss...

In my mind what I want to do is to share the remaining 200GB of the disk flexibly between ESXi for virtual machines and Network shares (Windows/Linux). Would this be iSCSI storage? and how would I go about this? I've seen that there are many tutorials on setting up iSCSi, but I'm not really sure whether I am way off-target with the concept in what I think I want to achieve?

I'm a relative newbie to VMware and have been reading about iSCSI targets, intiators etc.

Finally, how does this scale when I add several more disks and want to create a ZFS RAID set? Do I start from scratch?

I appreciate any input/insight you can provide.

Tim.

Answer 1

You can use iSCSI for this, it would allow for easy migration of storage off this physical box later, if you choose to do so. On this stage however, you can export physical storage to your VMs, which would reduce complexity of your setup. You wouldn't turn your single disk/partition ZFS pool into raidz, however you may add disks later, create raidz pool of them and zfs send/zfs receive snapshot from single disk pool to raidz pool.
As I imagine it now, you have a partition to hold your FreeNAS VM. Then you'd create another partition and attach it as a virtual hard drive to FreeNAS VM, create a ZFS filesystems on it and export as iSCSI/NFS/CIFS shares.
What you want to do, however, to use ZFS data health features, is to migrate this filesystem to physical disks as soon as possible.

Rough overview of this migration:

1. On actual, partition held ZFS filesystem create a snapshot:

   zfs snapshot datapool/data@migration

2. From new disks create raidz pool. Remember, that you can't add another device to raidz setup, but you can add another raidz to the pool itself later:
   

   zpool create datapool2 raidz2 c4t0d0 c4t1d0 c4t2d0 c4t3d0

3. Send/receive the snapshot you created, to migrate data:
   

   zfs send datapool/data@migration | zfs receive datapool2/data

To understand this better, read this blogpost.

Answer 2

You did mention wondering if you were off target - been a while since you posted so maybe its all sorted. I've got an N40L which came with a single 250Gb drive and 2Gb of memory. It has three additional slots for 3.5" internal hard drives.

I'm only part way through my journey but here are the key points I've picked up along the way.

1. N40L needs more memory - I brought the N40L up to 8Gb (2 x 4Gb sticks); I opted for the full spec (ECC RAM) but plenty of people show that it will work with cheaper non-ECC and some show that you can exceed the HP specified limit and put up to 16 Gb (2 x 8 Gb sticks). I'm being conservative throughout this exercise because I want fewer variables in play; when I get my NAS in place I want reliability above all. I had trouble installing esxi 5 (HP's ISO) until I did this upgrade - sounds like you've got that covered if esxi is up and booting off your flash drive

2. After playing around with esxi a bit I decided to run freenas in a vm on the N40L. There are other options like openindiana, napp-it etc. I'm sure there are pro's and con's here but it seemed to direct me straight to the result I'm after. The real issue that came next was - what disks and how to get them as cost effectively as possible - again want a low risk option. Fact, the disk that came with the machine is a Seagate Barracuda 7200RPM - there is a lot of discussion on which disks are best - it sounds like WD Red drives are most suited to NAS, but I haven't gone to that expense. The appeal of ZFS RAIDZ is that I should be able to recover from a single drive failure. Again, HP says maximum 8 Tb (4 x 2 Tb), but plenty of posts around bumping this up too. I was going to try 3 Tb drives, but landed on 2 Tb drives with the assumption that once these are in, with ZFS, I can iteratively replace them with 3 Tb if I need to grow.

3. I got my drives at Office works - they were in Seagate expansion drive enclosures (STBV2000300) - why it is cheaper to buy this way is a whole topic in itself. I ran SEATOOLS long generic tests before cracking open the cases - a few tabs, a few screws; inside there's a perfectly good STDM2000DM001.

4. My current understanding is that to properly run ZFS the drives should be directly available to the NAS VM - this is called RDM. This is a bit fiddly on the N40L but so far has worked just fine. Take note of the individual serial numbers of the drives as you put them into the bays - the screws are in the server case door. Then from the esxi command line (I have a kbd and monitor on my host and use alt-Fn keys to login as root) - cd to the existing vmfs datastore you have on the 250Gb drive and create these vmdk files for the other 3 drives to be RAW mapped. This post gave me the command: http://cyborgworkshop.org/2011/01/08/enabling-raw-sata-access-in-esxi-free/ however I couldn't run fdisk for some reason, so you can find the disks using ls -l /vmfs/devices/disks Here's the command: vmkfstools -z /vmfs/devices/disks/t10.ATA_____ST32000542AS________________________________________5XW205BS RAW-2TB.vmdk -a buslogic

5. I then followed the basic install for freenas 8.3 for esxi adding these drives to the FreeBSD vm I created by editing the vm settings and adding 3 hard disks as Mapped Raw LUN on scsi ports 1:0, 1:1 and 1:2. Again plenty of posts around that describe this. Freenas found the new drives no problem.

That's as far as I've got. I plan to test this more before I cutover - am particularly interested in booting the vm with one of the drives removed to see what happens

Answer 3

Quick how-to: FreeNAS RAIDZ on VMware ESXi

This took me ages to figure out, but it's actually quite simple to do.

This assumes you want RAIDZ1 and have three physical drives for the virtual disks that FreeNAS will use. This also assumes you already have VMWare ESXi set up and running.

• Create three datastores in ESXi, one for each of three three separate physical disks you have installed in your NAS box
• Follow instructions in FreeNAS manual to get the VM up and running...
• Create a 4GB VM
• Edit the VM settings and add 3 virtual disks of 100GB, put one on each of the 3 datastores.
• Boot VM from FreeNAS ISO and install on the 4GB VM (may appear like it's the ESXi flash drive on the server it's found (if you're running ESXi off a flash drive like I do), but it's not)
• Once installed and rebooted the disks should be available in the FreeNAS GUI.

To create a RAID set in FreeNAS:

• Volumes > Volume Manager
• Add the three disks and select the ZFS RAIDZ options
• Set the permissions: leave at Unix ACL, and tick all the Read/write/Execute options (yes, not good for security, but this is for testing)

Create a Share:

• Windows (CIFS) Shares
• Add Windows (CIFS) Share
• Path: choose the volume created above
• Allow guest access