Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm building an virtual environment for a small business. It is based around a single ESXi 5.1 host, which will host half a dozen or so VMs. I'm having some doubts regarding how to implement the storage though. I naturally want the datastore to be fault tolerant, but I can't get the funds for a separate storage machine, nor expensive hardware RAID solutions, so I would like to use some software RAID (lvm/mdadm, most likely). How can this be implemented? My only idea so far would be to create a VM which has the storage adapter as passthrough, puts some software RAID on top of the disks and then presents the resulting volumes "back" to the ESXi host which then creates a datastore from which other VMs get their storage presented.

This does seem kind of round-about, do I have any better options? From my research, passthrough seems to come with quite a few drawbacks, such as no suspend/resume etc.

share|improve this question
1  
This would be a sure-fire way to kill your I/O performance. Aside from the fact that you create dangerous dependencies. And aside many other things. Do your future self a good deed, and forget about this. Fast. –  Roman Jun 30 '13 at 11:03
1  
@Roman: Interesting, why would it be bad for performance? But I understand the dependency issue, and I'll throw this idea out. Thanks. –  carlpett Jun 30 '13 at 11:15
1  
Well. Take pencil and paper. Assume a VM wants to write a block. Try to trace the way the block takes through ALL involved subsystems before it really lands on a physical disk. Mark where locks, and their type, are involved. Mark which subsystems are emulated, not real. Spoiler: You may abort when overwhelmed or have realized that it's really not a good idea (which should be fairly soon). –  Roman Jun 30 '13 at 11:27
    
Except that in actual fact, although the scenario may be complicated, in reality the write is far more direct as each 'emulated' stage is optimised to pass through. It is this same kind of talk that suggested that virtualisation itself was not worth doing because of the performance impact. Don't just assume it is not possible. –  JamesRyan Jun 30 '13 at 11:53

4 Answers 4

ESXi is not a general-purpose OS and shouldn't be considered so - stick absolutely to the Hardware Compatibility List and therefore use a tested and approved hardware RAID adapter. Choose to do anything else and you'll join the swathes of others who cut corners and end up back here complaining that their systems don't work or they've lost data - we get LOTS of those.

share|improve this answer
    
All the hardware including SAS controller is on the HCL, but I guess this could extend beyond what is done for compatibility testing... –  carlpett Jun 30 '13 at 11:07
5  
+1, if you really want to use software raid at the hypervisor host then ESXi isn't the right product. Like @Chopper3 said, get a general-purpose OS like Windows Server 2012 or Linux with kvm. –  pauska Jun 30 '13 at 13:50

I would fire you if I were a small business and you deployed something like this... This is a common theme, though. VMware has a well-defined hardware compatibility list. However, when used as a standalone server, you NEED hardware RAID. Non-RAIDed disks will work as well, but that's not what you want. So my questions:

  • Not enough funds for storage? What type of server hardware is this? You can afford disks but no RAID controller? A compatible RAID controller is not expensive.
  • Isn't this a case of managing customer expectations? Separate storage would clearly be more expensive than hardware RAID.
  • While all-in-one storage solutions are possible, they are best for specific technical requirements, not cost-cutting.

It's a case of VMware-abuse. Software RAID is not supported. I would go back to the customer and revise the build/requirements.

"How much is your data worth?"

share|improve this answer
1  
I must say I've been quite surprised at the aggressiveness around this question. I thought I made it quite clear that I didn't really feel comfortable with this idea and wanted some insight if it was a feasible way forward. Apparently it is not, so I discarded it, as I noted in the comments above. I further started to write a meta post trying to understand why it seemed to evoke such hard feelings, when I found this: meta.serverfault.com/questions/5337. Practice what you preach? –  carlpett Jul 9 '13 at 19:31

Check out this blog for an example of using ZFS as a backing store http://blog.laspina.ca/ubiquitous/encapsulating-vt-d-accelerated-zfs-storage-within-esxi

That being said the warnings about using standard hardware are valid. ESXi is quite picky.

share|improve this answer

I think that your idea could work quite ok. See, for example, https://communities.vmware.com/message/2368577#2368577

It's a pity that ESXi does not support mdadm. So, your idea sounds like round-about, but I assume, that if you will configure everything properly, there will be only a very minor performance overhead. See the following article about the performance: https://www.linux.com/news/hardware/servers/8222-benchmarking-hardware-raid-vs-linux-kernel-software-raid

If you have a small setup, mdadm will probably use only 5-25% of single CPU core. From my own experience, I had a very low CPU usage by mdadm on CentOS server with RAID 5, which was connected to ESXi as NFS share over 1 Gbit network. However, we had a problem with VMs that used disks heavy (mainly not because of software RAID, but because of NFS). Please see my another answer for details: http://serverfault.com/a/608120/118677.

Another advantage of such setup will be that mdadm is well documented, it is easy to reconnect the array to another server in case of server failure.

However, you should consider that the usual setup of ESXi does not expect the datastore to be on the same server with ESXi itself. In your case, if hard drive with ESXi and/or "VM which has the storage adapter as passthrough" fails (and it is not in RAID, of course), your datastore will not be accessible anymore. If you have separated datastore, you will need less steps to restore your setup in the case of failure. So I consider that you should try again to find funds for separate datastore PC. It could be a used PC with 1-2 GHz CPU and SATA controller, where you could setup Linux OS with mdadm.

P.S. Do not forget to setup monitoring (e.g. e-mail notifications) about the status of your mdadm RAID array.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.