What storage connection method should one prefer to use for connecting ESX servers to a shared storage server with 10GbE links?

Speficically, I have 2 ESX servers for VMware ESX and one server for shared storage.

The storage server is 2 x Xeon E5504 2GHz, 24GB RAM, 12xSSD + 12xSATA and battery backed RAID. The ESX servers are much the same but with 2 small SAS drives.

All servers have 10GbE adapters connected like so: enter image description here

I have a licence for ESX 3.5 but for testing purposes I am currently running ESXi 4.1. The storage server runs Windows 7 for testing purposes.

I am aware about at least 3 methods:

 2. NFS
 3. FCoE

Which one would you recommend to choose and why?

  • It may be helpful to know what brand of storage or what OS you are using for your storage server. Protocol capabilities can vary from vendor to vendor. – Cakemox Mar 30 '11 at 15:47
  • @Cakemox : He said "The storage server runs Windows 7 for testing purposes." – mfinni Mar 30 '11 at 16:36
  • It's Supermicro 2U 24 bay server with two Supermicro AOC-STGN-i2S 10GbE NICs. The os is Win7 right now but can be anything. – Henno Mar 30 '11 at 17:16

No 'if's, no 'but's - if you have the option of using 10Gbps FCoE and your configuration has proven stable then it's the best and only way to go.

It's still quite new but the efficiencies are overwhelming in comparison to iSCSI, and NFS is just plain 'different'.

Be aware however that you should be right up to date with ESX/ESXi 4.1U1 for best FCoE performance/stability and the list of supported 10Gb NICs/CNAs is quite limited but other than Infiniband systems I've never seen shared performance like it. I'm currently moving all of my FC to FCoE, though this won't be complete for over a year due to the volumes involved.

  • agreed if you have FCOE available there is no thought involved, you implement FCOE – Jim B Mar 30 '11 at 17:23
  • I found a whitepaper about ESX 3.5 10gbe support: vmware.com/pdf/10GigE_performance.pdf. It says that ESX 3.5 U1 is fine. Is it really? – Henno Mar 30 '11 at 17:25
  • 10Gb yes, FCoE over 10Gb no. – Chopper3 Mar 30 '11 at 17:39
  • @Chopper3 : Could you refer to some facts which state that FCoE is better than NFS and ISCSI? – Henno Mar 30 '11 at 17:42
  • 1
    It does, you're correct. – Chopper3 Apr 5 '14 at 8:11
NFS   - is file level storage and is the slowest - routable 
FCoE  - best performance but only if you use it locally into 
        a stub network (is not routable)  
ISCSI - very good performance but adds a bit of complexity - 
        on the flip side is routable

If your goal is ease of use, you may want to consider NFS. It has a mediocre performance overhead (-~5% overall throughput, +~20% storage-related CPU) when compared to the FC.

Here's a comparison of NFS vs iSCSI vs FC in a 4Gb and 10Gb environment:


  • FCoE is very different to FC though Hyppy – Chopper3 Mar 30 '11 at 17:53
  • FCoE wraps FC in an ethernet frame. Besides some lower-level optimizations regarding latency handling to mimic native FC behavior, I don't see what the significant difference is. FCoE doesn't outperform FC though, so the comparisons are still valid from a baseline perspective. – Hyppy Mar 30 '11 at 17:59
  • I'm wondering how much CPU NFS FCoE and ISCSI actually consume. In absolute percentage. – Henno Apr 1 '11 at 17:51

I am not sure how this would apply to your environment, but I can share with you how ours is currently configured.

First, we are 100% iSCSI for the shared storage and we are running it over redundant dedicated 1GB network links. We currently have 4 ESXi hosts inside of an IBM BladeCenter chassis.

Each host has 2 internal drives set in a mirror array that stores the base ESX installation.

These hosts all share centralized ESX storage on a pair of HP LeftHand iSCSI SAN nodes which reduced the complexity of SAN management as the SAN OS on them handles the actual drive allocations internally for each LUN and moves them around to provide maximum redundancy as the SAN cluster itself changes.

Simply, we went with iSCSI because it was the default method for our new SAN nodes, but I couldn't imagine taking any other approach at this time. We have had 0 outages over the last few months and are running our entire SAP infrastructure (DEV/TEST/PROD and the related databases) all on these 4 machines.

Hopefully, this provides a little bit of insight into another architecture utilizing iSCSI and ESX.


I've no experience using Windows 7 as a storage server, but I have a small VM environment that I use NFS backed onto Solaris, and that works really well. Very easy to set up and configure and the performance is quite good. I've got some iSCSI shares through COMSTAR on there as well, and the performance is similar.

Cakemox is correct though, just because I get good results on NFS talking to Solaris, doesn't mean that NFS is right for a Windows based solution. You might find better support in the iSCSI realm.

As for FCoE, I didn't think that was supported by vSphere/ESX?


I doubt that this affects your Win 7 testing storage server, but if you're going to replace the storage server with a different dedicated SAN or NAS, check the capabilities of your vendor and how that will affect your storage presentation.

For one vendor (EMC in our case), they recommended the VMWare servers use NFS to access the storage, rather than iSCSI. Exposing the VMs via NFS lets the SAN/NAS hardware perform deduplication and thin provisioning. iSCSI to VMFS datastores on the same hardware would prevent the SAN/NAS features from running.

  • 1
    Sorry but you can both dedupe and thin provision using FC, FCoE, iSCSI or NFS - there's nothing inherently better about NFS for those things. – Chopper3 Mar 30 '11 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.