Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm building VMs in my lab and I want to replicate the situation that my fileservers will be encountering in production. Here's a brief overview of what I want to do.

I have multiple ESXi 4 servers. They are accessing a SAN, and each virtual machine will exist in its own LUN on the array. I also have LUNs which house data that needs accessed by the VMs.

My goal is to have a VM (lets call it fs, for fileserver) started on VMhostA. I want fs to be able to access a data LUN in raw mode, meaning that if fs goes away, I want to be able to mount it as ext3 (or whatever) from a physical (non-virtual) machine. This means that the data on the LUN must not be housed in a vmdk file. In addition, I want to be able to use VMotion to move that VM to VMhostB (and obviously, maintain access to the data LUN)

It is my understanding that I need to present the LUN as a Raw Disk Mapping (RDM)? From what I have read, the RDM file created can be stored with the fs virtual machine or on another datastore. Would it be correct to assume that it should be stored with fs?

Also, there exists two types of RDM, it seems. There are physical and virtual modes. I have encountered conflicting documentation from various sources, so I'm not sure what to think. Does it matter which I select in this case? What are the differences, as they apply to my situation?

Thanks very much for reading all the way through ;-)

share|improve this question
up vote 3 down vote accepted

I don't have direct experience using RDMs in vSphere 4 ESXi (only in ESX 3.5), but everything you're saying re: what you want to accomplish with presenting your data LUN via RDM jibes with my knowledge. The mapping file is typically stored in the datastore with the virtual machine using it, as you surmise.

re: virtual RDM mode versus physical RDM mode - The background I'm finding seems to indicate that in a virtual/physical cluster you'll want to use physical mode. Having said that, I'm also finding documentation that indicates that nothing is written to the LUN other than the guest filesystem information even in virtual mode (see I would suspect that, since you're looking for the capability to mount the RDM LUN "cold" on another machine (i.e. not a true cluster application) that virtual mode will work fine for you.

Since virtual mode seems the most flexible, and gives you the most potential to use VMFS features on the LUN, I'd suspect that you'll want to try using virtual mode first.

My recommendation would be "test it out and see how you like it". Obviously, you being Matt Simmons, you're already doing that... >smile<

share|improve this answer
Thanks :-) Yeah, definitely testing and playing with things. Thank goodness they provide ESXi trails! It's just always disconcerting to be faced with a corner case that you didn't anticipate during testing, so I thought I'd see what the more knowledgeable community thought! Thanks very much for the answer, Evan! – Matt Simmons Jun 29 '10 at 21:26

RDM's are definitely what you want to use and what you have outlined will work the way you intend.

The choice between Virtual and Physical mode depends on use case, and as you indicated Virtual Mode is the most flexible. Physical compatibility mode is only required when you need to use software within the Guest that requires direct access to the disk hardware - SAN management software for example. It's also required for mixed physical\virtual clustering like MSCS, probably for the same reason - I suspect (but I'm not certain) that all nodes in the cluster need to be able to talk directly to the same shared hardware in the same way. Physical compatibility mode has no effect on standard vMotion but it does prevent Storage vMotion, Snapshotting, cloning or Conversion of a VM into a template. All of those restrictions stem from the fact that a physical mode RDM can't support VMware snapshotting techniques.

An RDM in Virtual mode behaves exactly like a disk presented via a VMDK - Storage VMotion \ cloning \ Snapshots etc all work.

As to your other question about the files involved, an RDM is made up of two parts; a stub\mapping file (ending in xxx-rdm.vmdk) that contains meta data for the volume, and the actual raw LUN that the stub points to. The mapping file can be stored on any VMDK (or NFS share) visible to the ESXi host(s) but the most sensible place to put it is with the main VM files because it is really just a small text file (a few kB). Snapshot delta files will end up in the same place if you choose to use those and have selected Virtual Compatibility Mode so you should factor in capacity for those if you are going to do that.

share|improve this answer
That's an interesting (and important!) point about the delta files. Thanks for your answer! – Matt Simmons Jun 29 '10 at 21:29
Although the main questions are answered, I wanted to add a few comments since I recently had to deal with MSCS using RDMs and upgrading from 3.5 to 4 (in the hopes it will save someone all the research I had to do). If you ever need to have a second VM access the RDMs as well, the mapping file needs to be accessible to both VMs (ie, on a shared LUN). This wasn't the case in 3.5, where each VM could have its own mapping file(s). In 4, the first VM creates the mapping files, and any additional VMs need to have the mapping files added as existing virtual disks. – Paul Kroon Jun 30 '10 at 2:44
@Paul - great comment. – Helvick Jun 30 '10 at 18:32

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.