Note: Some of you may ask, why not just use a cheap hardware solution. The solution is not so cheap in the country it has to be installed in, with as much as 100% import tax on high end technology. It's much cheaper to get a couple of new components in and use an existing server than to try to get a new server imported in.

Note 2: Please excuse the confusing FC and iSCSI, I see them as the same thing from an OS perspective. So either solution works.

I intend to build myself an "as cheap as possible" SAN with Fibre Channel. I'm less concerned about HA at this point, but may need to do HA down the road. The idea is that I need a decent iSCSI solution for VMWare vSphere to run virtual machines from.

I've been running vSphere from local storage, and some applications (like SQL Server) just work better with native drives rather than running on VFS.

My thought was that I could take a server I alrady have and plop in an FC HBA then run some software that will allow me to treat the drives as iSCSI targets (there are several choices there). And, if I use standard SATA drives, or possibly even SAS "near line" drives i can get a lot of storage for a lot less money, and can maybe get near 15K SAS drives running in RAID5 by running the slower drives in RAID10 and still save money.

However, I do have a few questions that I hope someone can answer as i'm not an FC guru:

1) Do I need an FC Switch if i'm just running between two servers? Can I do Point-to-Point with any two FC HBA's? Or does the HBA have to support that mode explicity?

2) if the HBA's have 2 ports on them, can i run 2 cables and get 2x performance seamlessly (Teaming)? Or do I have to assign some LUN's to one port and some to another?

3) How much CPU load does handling SAN requests take (obviously depends on the OS, but in general)? Can I take a lower spec server and put in a decent SATA controller with room for drives and an FC HBA and not really worry about CPU load much (assuming it does nothing else)?

4) Is this even a good idea?

5) Would GigE make more sense with multiple NIC teaming? I mean, could I put in a couple of 4 or 8 port E1000's and approach a dual port 2 or 4Gb FC in terms of performance?

  • are you saying that its cheaper to get a bunch of components and build a san from scratch, then it is to buy a complete device that has the exact same specifications? So you could buy case(s),motherboards,drives and such? – tony roth Aug 31 '10 at 0:58
  • No, I'm saying that I already have a server I can repurpose. So in order to build a SAN i'd only have to invest in some drives and some connection technology between the two servers (FC, GigE, etc.). And, because of the high import taxes, I am not paying more tax on stuff I already have. – Erik Funkenbusch Aug 31 '10 at 1:01
  • Cheap and SAN, two words that should never be uttered in the same sentence. – Tom O'Connor Oct 10 '10 at 15:00

You should probably just use a FC SAN; if you have FC Storage already, you can grab an FC Switch and FC HBAs for the host servers; that will be enough to get a simple FC SAN going.

You can use a server to act as a FC-iSCSI gateway; software like Starwind or SanMelody run on Windows; or most Linux distros come with iSCSI Target software.

Handling the SAN IO will depend on whether it's IOps or MBps you're looking for. More IOps requires more CPU usage to process each; both require as much cache and the fastest buses you can find.

GigE is just as fast as FC; but there's more overhead with GigE usually. iSCSI runs on TCP/IP and that adds a lot of overhead. The Ethernet frame is very comparable to the FC frame however; so technologies like ATAoE and HyperSCSI can keep up, with the caveat that a single 4Gbps FC can do a single stream at that speed, and 4 GigE would need 4 connections to saturate the fabric (theoretically; real life is always a little worse).

So you talk about SATA and SAS drives... do you have a FC to SAS controller that you're planning on plugging into a FC-iSCSI Gateway Server, that's then relayed to the Host servers?? That's a lot of overhead and will pretty much kill performance.

If you want to use SAS hardware you can create a SAS SAN cheaper yet (and no FC or iSCSI); drives plug into it, SAS HBA on the servers, and the SAN chops the drives into "LUNs". Products like the HP MSA2000sa will do this. This is more limited than FC or iSCSI; but for simple environments is very cost effective.

  • Ok, that's just my ignorance showing. I know you get LUN's in both FC and iSCSI, so I consider them more or less interchangable from the servers perspective. Can you elaborate a bit more on the SAS SAN? I have a Dell 2900 2U server that has a PERC/5 SAS RAID controller. This has worked ok, but I would really like to give better performance to the virutal machines than running virtual drives over the VMWare VFS filesystem. I want to give the VM's "native" like performance by assigning SAN LUN's. – Erik Funkenbusch Aug 30 '10 at 20:39
  • I looked at the HP MSA2000sa, but i don't really consider that to be "cheap". Looks like a base cost for an enclosure is in the $2500 range, which would translate to a $5000+ price tag to get it imported into the country it needs to be in. As I said, I already have a server I can repurpose for this, but need a semi-wide bandwidth connection to make it worth it, that's why I looked at FC. – Erik Funkenbusch Aug 30 '10 at 20:43
  • Also, no. I do not currently have FC storage. This means the solution has to be built from the ground up, which is the reason i'm asking about connecting directly (no need for a switch because it's just one server to another) – Erik Funkenbusch Aug 30 '10 at 20:50
  • I don't normally deal with SANs that cost <$20k. So that box seems pretty cheap from my prospective. I don't know of any software or hardware for a server that will allow a SAS SAN (it might exist; I just haven't heard of it). Also, if you're using fixed size VFS, your performance should be close to passthrough LUNs. You can also use passthrough without a SAN, but you wont get HA, as you'll only be able to use LUNs on the host server. A LUN is just a Logical Disk, sometimes they're real disks, other times there's a middle device that presents a LUN abstracted from the physical disk parts. – Chris S Aug 30 '10 at 20:51
  • Yeah, I know.. most people that need a SAN can afford to get the premium solutions. I'm not in that boat. This is where my thoughts on iSCSI came in, because there are products like Starwind and Linux that can offer iSCSI target functionality. Seems there should be a way to do that over FC as well. – Erik Funkenbusch Aug 30 '10 at 22:18

If you are wanting to do iSCSI, you don't need FC hardware.

You may want to look at OpenFiler or FreeNAS to build your storage system.

[UPDATE] It really appears you are looking for basic direct attached storage. A homebuilt version of a Dell MD1000. iSCSI is irrelevant in this case. SATA, eSATA, SCSI, SAS, and Fibre Channel are all possible protocols. This isn't shared storage, but just expanded local storage.

  • I'm not wanting to do anything specific, just whatever works out to be the best value solution. Best bang for the buck. I want to add 5TB's of storage to an existing datacenter that has only the need to connect the SAN to a single Dell 2900 server. If Ethernet cheaper great and gets similar performance, great. If FC is the best value, then i'll use that. – Erik Funkenbusch Aug 30 '10 at 20:47
  • We use a very expensive SAN over gigabit ethernet connecting 4 physical servers, with 15+ virtual servers. We have seen excellent performance. – Craig Aug 30 '10 at 20:51
  • Well, the way I look at it, each SATA drive is capable of 3Gb/s (yeah, i know it will be lower in real use.. but i'm looking at peak here), and if there are 5 LUN's in use, that's a theoretical 15Gb/s. FC 4Gb will obviously push 4Gb/s, and if it's possible to team them maybe 8Gb (ignoring overhead, which FC has less of than Ethernet). I don't see how GigE can provide high enough throughput especially consider it's higher overhead. And that's not even talking about SAS drive performance. – Erik Funkenbusch Aug 30 '10 at 21:15
  • Remember, my solution is attempting to provide better performance for VMWare drives than local SAS 15K drives running the VM's on VFS. VFS adds a lot of overhead, and the translation to storage files adds overhead. Native LUN's reduces that a lot. – Erik Funkenbusch Aug 30 '10 at 21:18

Don't confuse iSCSI and Fibre Channel, they are different, even when having comparable purpose.

iSCSI works completely over IP on standard ethernet components and doesn't need any special hardware at all. Fibre Channel on the other hand connects storage devices over a specialized (optical) network with a totally different protocol than IP.

If you can use iSCSI targets for storing data, you don't need Fibre Channel at all as you could connect many disks with SAS or SATA (preferably with a hardware raid controller) and then export the storage as an iSCSI target.

  • Point taken. I simply consider them equivelent from the OS perspective, both iSCSI and FC provide LUN's that can be allocated to VMWare VM's. – Erik Funkenbusch Aug 30 '10 at 20:45

You may also want to look into AOE, ATA Over Ethernet, from coraid.com. This is layer 2 on GigE works well because there is no IP overhead, it's a little faster than ISCSI. You can either buy a coraid shelf that has multiple ethernet ports or use their free software to present a disk from a linux box. The client software load balances traffic on all ports. They also have a supported card for VMware.


Cheap, good, fast; pick any two.

You can build an iSCSI target using something like FreeNAS or a couple other solutions, but have you considered using NFS as your shared VMWare storage? Again, FreeNAS or Nexenta can do this easily. Just use a hardware RAID controller to get the performance you need or in the case of the nexenta, have enough CPU in the storage box to do the Raid.

With a ZFS based solution you also get deduplication on the primary storage so you save a lot on space. NetApp is the only other storage that can do that.

  • Well, i'm looking for Cheap, fastish, and goodish.. and by goodish, I mean, no HA, no Support contracts, no extended warranties, no high end equipment, just cheaper prosumer stuff. Basically, not premium hardware (SATA or Nar line SAS drives, cheaper HBA's, etc..) – Erik Funkenbusch Aug 30 '10 at 22:11
  • I really don't want to use a filesystem over ethernet. I want BLOCK addressable storage, so that's why i'm not looking to use NFS or Samba or anything else over a network. I'm not sure how I can use ZFS when VMWare would want to use VFS, nor do I see how de-duplication would help since we're talking virtual disk files on the filesystem (double performance hit) – Erik Funkenbusch Aug 30 '10 at 22:15
  • For block storage, iSCSI will be it and right, de-dupe will not help here. But, you really should check out using NFS, VMWare has excellent support for this in vSphere (ESX 4.x). Being on the cheap you are looking at iSCSI software initiators, so CPU will be similar with either one. I think you might be confusing the filesystem layers when using NFS. VMWare doesn't see ZFS, as this is the underlying storage, managing the raid sets, de-dupe, and error correction. VMWare sees the NFS "share" (think Windows shared folder). VMFS files are a shared FS and de-dupe works great with them. – JGurtz Nov 15 '10 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.