I'm replacing two old rack servers with a new one that has plenty of power to take over the functionality the current servers. The server is a 4U rack mount with 16 3.5" SAS drive bays, two 2.5" bays, a Xeon E3-1230v2 CPU and 32GB of ECC RAM.
My issue is the following. I would like to have a FreeBSD file server with ZFS managing disks. However, I need other VMs for e.g. a shell/git server, mail server etc. I'm wondering how to deal with the following issues:
I want ZFS to fully manage the disks, so I'm not using any hardware RAID. Should I pass the SAS controller directly to the FreeBSD system as passthrough PCI? Maybe just the disks one by one?
I want to maximize the reliability of the setup. On what disks should I install the hypervisor and where would you put the VM system images?
For (2) I have the option of having a RAID setup on the SAS controller and using that as system disk to store the hypervisor as well as VM images. However, this makes PCI passthrough to the file server impossible. Another option is using the two 2.5" bays.
Recommendations? I have yet to choose which virtualization platform to use, but this should not have much of an impact.