Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm contemplating the next restructuring of my medium-size storage. It's currently about 30TB, shared via AoE. My main options are:

  1. Keep as is. It can still grow for a while.
  2. Go iSCSI. currently it's a little slower, but there are more options
  3. Fibre Channel.
  4. InfiniBand.

Personally, I like the price/performance of InfiniBand host adapters, and most of the offerings at Supermicro (my preferred hardware brand) have IB as an option.

Linux has had IPoIB drivers for a while; but I don't know if there's a well-known usage for storage. Most comments about iSCSI over IB talk about iSER, and how it's not supported by some iSCSI stacks.

So, does anybody have some pointers about how to use IB for shared storage for Linux servers? Is there any initiator/target project out there? Can I simply use iSCSI over IPoIB?

share|improve this question
1  
Are all the hosts physically close to each other? –  chris Aug 10 '09 at 17:25
    
@chris: yes, all in a couple of racks –  Javier Aug 10 '09 at 17:27

8 Answers 8

I've just had to deal with an IB SAN using Mellanox NICs. Works out of the box on RHEL

share|improve this answer
    
what did you use for storage target? –  Javier Aug 10 '09 at 18:05
    
Can't remember - too many systems come my way :) –  dyasny Aug 10 '09 at 19:18

Although it is possible to run iSCSI over InfiniBand via IPoIB, the iSER and SRP protocols yield significantly better performance on an InfiniBand network. An iSER implementation for Linux is available via the tgt project and an SRP implementation for Linux is available via the SCST project. Regarding Windows support: at this time there is no iSER initiator driver available for Windows. But an SRP initiator driver for Windows is available in the winOFED software package (see also the openfabrics.org website).

share|improve this answer
    
thanks for the pointer. do you (or anybody here) have first hand experience with those packages? what are the pros/cons of SRP relative to iSER? (besides windows compatibility, which is a total non-issue for me) –  Javier Dec 3 '11 at 18:46
    
An advantage of iSER is that is possible to define multiple targets on one iSER server. An iSER initiator can choose which iSER targets to log in to. SRP on the other hand is a host-to-host protocol: all LUNs defined on the target become available to each initiator - unless LUN masking has been configured on the target. Another advantage of iSER is that it is possible to configure password-based authentication. And a big advantage of SRP is significantly lower latency - that is because the SRP target implementation runs in the kernel while the iSER implementation runs in user space. –  user251384 Dec 5 '11 at 20:17

Do you need IB's latency benefits or are you just looking for some form of combination of networking and storage? if the former then you have no choice, IB is great but can be hard to manage, FC works great and is nice and fast but feels a bit 'old hat' sometimes, iSCSI can be a great solution if you consider all the implications. If I were you I'd go for FC storage over FCoE via Cisco Nexus LAN switches and a converged network adapter.

share|improve this answer
    
i want it just for storage, latency is not as critical as bandwidth. FC is the usual answer (besides iSCSI, of course), but I haven't really tried either, so I'd like to know if FC has any real advantage. What I like about IB is the cost/bandwidth ratio (best of any other option), and that it's included on several mainboards (unlike FC or 10gE). what kind of troubles make it "hard to manage"? I've found that even Ethernet can be a headache if you don't have really good switches. Is IB worse that that? –  Javier Aug 10 '09 at 19:07
1  
If you just want the bandwidth then IB is great, but most IB storage is actually just FC storage dressed as IB and iSCSI is slow for most things without 10GB and some form of QoS'ing. –  Chopper3 Aug 10 '09 at 19:14
    
and what about software storage targets? IET works great for iSCSI, but don't know if IB would need anything else, or if it's simply a matter of getting IPoIB and iSCSI over that. –  Javier Aug 10 '09 at 19:51

What about 10gb ethernet? The more exotic the interface, the harder time you're going to have finding drivers and chasing away bugs, and the more expensive everything is going to be.

Okay -- here is a cheap rundown given that everything's all within cx4 cable distances (15 meters):

(I'm using us dollars and list prices found on web pages. I'm assuming the vendor prices are USD are as well)

Is infiniband that much cheaper?

(please note -- I've never actually used any of this gear, I'm only going by whatever pops up on google after 30 seconds of googling. I'm certainly not endorsing it or making recommendations that it will do anything good or bad)

share|improve this answer
    
yeah, it's the simplest answer, but still far more expensive than IB, both for host adapters and switches. giving one to every server and storage box is out of the question for a while. –  Javier Aug 10 '09 at 17:14

NFS over RDMA works GREAT in Fedora linux

It's very easy to set up. Install the right rpms and tweak a couple of files. Just google to find instructions.

I used Mellanox MT25208 PCIe-x8 infiniband cards, flashed to the latest firmware. Total cost for two cards and a 15 M cable: $150. Who cares about "market adoption" at that price.

Smokin' bandwidth, well over 400 MBytes/sec, with very little CPU usage on client or server. The bottleneck is the RAID controller.

As a bonus, X over IPOIB is also smokin', you'd swear the app is local.

share|improve this answer
    
Also qpid messaging server sees ping times of 20 us. Yes that is 20 us. Unreal. –  user59577 Nov 8 '10 at 19:27
    
so it's NFS over IPoIB. glad to hear that works so well, but i'm not interested on NFS, as i do like sharing block devices for this. What about iSCSI on IPoIB? –  Javier Nov 9 '10 at 4:01

I approached the same problem by using 10-Gigabit iSCSI with a dedicated 6-port switch (HP 6400cl-6XG - $2200) and Intel dual-port CX4 NICs (Intel EXPX9502CX4 - $650). The cost per server came down to the NIC and a $100 CX4 cable. In this case, very little was needed to get drivers, etc. to work in a mixed Linux, Windows and OpenSolaris environment.

share|improve this answer

The difficulty with IB when building a SAN is to manage the srp target. There are very few pre-built solutions available and most are expensive. If products like Open-E introduced native IB support into their software (specifically srp) you would have an easy hardware solution. The client side is very simple to set up on RHEL and it works perfectly. We have a test system up and running now which is performing at 600MB/s consistently and under high load. The performance is amazing and the large amount of available bandwidth gives you great peace of mind and flexibility. Yes you are still limited to the speed of your array, but with IB you can connect multiple arrays without losing performance. Use one for backups, one for main storage etc etc and use them simultaneously without any loss of performance. In my experience, as a pure RDMA storage network, without IP, there is nothing that can beat IB and if you shop around, you can set something up for a very reasonable price. If someone was to introduce some storage appliance software similar to Open-E with full SCST SRP target support it would open up the mainstream market to IB and I for one would be very happy.

share|improve this answer
2  
whitespace... please –  Lucas Kauffman Apr 16 '12 at 6:13

I have not implemented an IB storage solution myself, but the main problem as I understand it is that the host drivers are not in wide use in your average environment. They are in wider use in the Windows world than in the Linux world. Where they are in use in the linux world, it's usually in "packaged" infiniband hardware appliances or supercomputing applications with tuned/customized drivers.

10g ethernet or 10g fiber is in much broader use.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.