Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

With only a modest budget, I want to move my 4 xen servers over to network storage -either NFS or iSCSI which will be determined based on how well it performs when we test it (we need good throughput and it must continue to work through link and switch failure tests). We may add another couple of xen servers at some point when this is done.

I don't know much about the design and operation of storage networks, so would really appreciate some hints from those with experience. The budget is around $3,800 excluding the storage appliance. I am currently thinking these are my options to remain on budget:

1) Go for used infiniband hardware and aim for 10gb performance.

2) Stick with gig ethernet and buy some new switches (cisco or procurve) to create a storage-only ethernet LAN. Upgrade to 10gigE later but try to use hardware capable of it where possible to reduce upgrade costs.

I have seen used, warrantied infiniband switches at reasonable prices (presumably because big companies are converging on 10gbit ethernet?) and the promise of cheap 10gb is attractive.

I know nothing about IB, so here come the questions:

Can I buy 2 x switches and have multiple HBAs in my xen and storage nodes to get redundancy and increased performance without complexity or expensive management software costs? If so, can you point me to some examples?

Do NFS and iSCSI work just the same regardless?

Is IB a sensible choice or could/should I use ethernet or FC on the same budget - I'm keen not to get boxed into a corner for future upgrades, however.

For the storage I am likely to build a storage server using nexentastor with the intention that I can later add more disks, SSDs and add another server to provide a failover option at the storage level. An HP LeftHand starter SAN is also under consideration, too.

Thanks in advance.

share|improve this question
You'll probably find that iscsi is more robust than NFS for this application. 10Gb ethernet is probably overkill unless you have a a justified need for it, as you can use multiple gigabit ethernet interfaces to increase throughput. Your budget is small for a storage solution. – Warner Mar 28 '10 at 17:45
Not an answer but a suggestion: Setup a test OpenFiler SAN box and connect it to a test Xen machine at 1Gbps. Configure a VM to run from it. This'll give you a much better understanding of how everything hangs together and what kind of performance you can expect. You'll then be in a stronger position to allocate the cash. Going in blind here could lead to a very expensive error in assumptions. – Chris Thorpe Mar 28 '10 at 20:58
up vote 2 down vote accepted

If possible, go with Infiniband. It is superior to 10g ethernet because it has small guaranteed latency. it is a much better technology for short distances - and breaks apart as real network over distances ethernet handles.

Regarding ibeing in a corner - you are. The corner is "server side cluster high performance network". The question is whether you CAN live with it - and personally I think it is a decent cornder to be in.

The big plus of Infiniband is not the higher bandwidth, but extremely low latencies. This is good for response/request throughput. Ethernet is a lot worse in the latency area.

Do NFS and iSCSI work just the same regardless?

If ou have to ask this question, you sould hire someone doing the setup. This is basic beginner stuff. iSCSI is SCSI over a network. Every IT-Tech should know what SCSI is (a protocol to handle hard disc communication), so it handles direct block read and write access.

Every guy having ever installed a disc knows that you have to partition and format this disc. NFS then sits on top of that (which you should know).

So, they are not the same and they come with significant differences.

Regarding what is better - it depends on XEN and what they suggest. Note that if you plan to run shared ISCSI (i.e. one ISCSI target, i.e. simulated disc, shared between multiple computers) - you can ONLY do that is the file system you run it on supports that. Most fiel systems do NOT - the driver will assume it owns the disc and noone else does writes. Havung multiple computes connect to one ISCSI target and use a non-shareable standard file system is a desaster running on a countdown - you are lucky to survive 5 minutes without data corruption if you do any writes.

share|improve this answer
Thanks for your answer, TomTom. I do know the difference between iSCSI and NFS (reading my question again it isn't worded very well). What I meant was are there any significant differences or difficulties in using IB as the fabric for these services versus over ethernet? From what I have read since, it seems that IB is designed for HPC and clustering more than data networking with these protocols. – StaringSkyward Apr 4 '10 at 20:30
IB. IB is cheaper (switches mostly - I am seriously surprised by the insanc pricing still in 10gbit switches) and has better latency. A LOT better. The disadvantages (shorter distances) do nto seem to hit you. IB also has higher bandwidth (12gbit, soon 24). – TomTom Apr 5 '10 at 3:39

I can only answer part of the question. iSCSI and NFS are fundamentally different beasts. NFS is a shared filesystem, and its basic building block is the file. iSCSI on the other end is a networked block device, and it is accessed in blocks, just like a local disk. On top of it you will add a filesystem. Generally you should think iSCSI just as SCSI (or FC), but over the network, i.e. one iSCSI disk belongs to one server. You can share iSCSI devices (as you can FC devices), but you have to use a cluster-aware filesystem like Oracle's ocfs2 or RedHat's gfs (I do not know about Windows, though).

If you plan to have shared data, then NFS is probably the best option, if you want to have something like a database, probably iSCSI is a better choice.

Hope this helps.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.