I have a VMware ESXi 4.1 server which I recently added 2x1TB SATA drives to. This machine runs a NexentaStor Community VM which hosts ZFS filesystems. Before adding the new drives, all the ZFS zpools resided inside virtual diska (VMDK files) which resided on the server's Vmware Datastore which is on a hardware RAID10.

The new SATA drives have no hardware redundancy, so my goal was to attach them directly to the NexentaStor VM and create a RAID1 zpool out of them.

I followed these instructions to create two Physical RDM files for the new SATA drives using vmkfstools -z /vmfs/devices/disks/idnumber RDM1.vmdk -a lsilogic

After adding the two RDM disks to the VM and creating a raidz1 zpool on them I started copying data to the zpool. The pool was taken offline and I was informed there were thousands of checksum errors.

I searched the web and found a number of people complaining of the same situation. (Example) I have since given up on using RDMs and have created two datastores and two 930GB VMDK files which I will place in a RAIDz1. However I want to know where I went wrong. A lot of people online said they have this configuration working.

My goals behind using RDMs as opposed to VMDKs were:

  1. Give the VM the ability to monitor S.M.A.R.T. status
  2. Allow ZFS access to the entire disk (since I knew these wouldn't be used for anything else)
  3. Make the drives easy to hotswap should one go bad
  4. Allow me to remove these drives and place them in another ZFS server should I need to

I had planned to use this same setup in a brand new ESXi 5.1 server which I will be setting up later this week. In that case #4 is particularrly important because I wanted to add an existing zpool to a new VM. Why did I get these checksum errors? Is it related to ESXi 4.1? Is there something I did wrong?

Edit: I have created the same setup with VMware ESXi 5.1 and have no issues so far. I'm going to test this extremely thoroughly but so far this appears to be an ESXi 4.1 issue.

  • But the right solution is still to use PCI or controller passthrough. – ewwhite Feb 7 '13 at 23:55
  • I'd still like to understand why that is. Can you point me to a document explaining what the proper usage of RDMs is? (Not to mention that I cannot use PCI passthrough for my SATA controller based on physical hardware configuration) – Josh Feb 8 '13 at 1:33
  • For example I have seen Chopper3 recommend RDMs in other serverfault posts. I'm looking for VMware documentation on when they should/should not be used. I'm trying to learn, looking for more than "don't do it". – Josh Feb 8 '13 at 1:42
  • RDM's are used in situations where they are necessary (large volumes, database clusters, cluster filesystems, SAN-backed solutions). Your situation would benefit most from a dedicated SAS controller. I've linked you to two examples in my answer. Also see here. But you don't have to take my word for it... – ewwhite Feb 8 '13 at 1:58
  • Thanks @ewwhite I will read that tomorrow. Also, please don't misunderstand, I'm not doubting you, I'm just trying to broaden my knowledge, that's all. No offense intended! – Josh Feb 8 '13 at 2:29

Is it strongly recommended to avoid using RDM's for this purpose.

Pass your SATA controller through to the virtualized NexentaStor instance using "PCI Passthrough". This requires a reasonably-current processor to do so.

See: Hosting a ZFS server as a virtual guest

That said, there's no major issue running ZFS on VMDK's if you have an underlying hardware RAID. Can you provide more information about the hardware solution?

  • What if I cannot pass my SATA controller through? For example, in the case of the new machine I will be setting up, I will need to install VMware ESXi to a SATA drive. – Josh Feb 6 '13 at 15:40
  • Well, what you're doing may not be possible. You could install ESXi to a USB drive, but you'll still need persistent storage to house the Nexenta VM. So in this case, you need another controller if you want full-disk access... OR a hardware RAID to house the full setup and run ZFS on a VMDK. – ewwhite Feb 6 '13 at 15:51
  • OK... I'll see what options I have. Can you explain why RDMs don't work like this or how others made it work? Specifically, I'm trying to understand how RDMs work and what they can be used for / why they failed in this configuration. – Josh Feb 6 '13 at 17:02
  • People don't use VMware RDM's for this purpose. – ewwhite Feb 6 '13 at 19:14

As far as I can tell, this was caused by a bug in the RDM implementation in VMware ESXi 4.1. Using VMware ESXi 5.1 I see no issues with RDMs at all. I'd still like to understand more about why the RDMs didn't work as I expected or what they are used for, but using RDMs on 5.1 has been very stable and allows me to achieve my goals.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.