1

I've installed xen-hypervisor-4.1 package onto an Ubuntu 12.04.3 server.

It has an infiniband adaptor installed and I run IPoIB with it.

What I'm wanting to do is provide the ipoib to one or more guests. However using linux bridge is not possible.

# brctl addbr br0
# brctl addif br0 ib0
can't add ib0 to bridge br0: Invalid argument

And that's because ib0 is not a true ethernet interface. I already have ipoib deployed on the network so eoib may not be a compatible option?

Also, if I was to simply pass through the Infiniband adaptor that would only work for a single guest.

How do I make the infiniband network available to certain virtual machines?

4
  • 1
    Probably you'll need to route it. Sep 11, 2013 at 2:04
  • I do have a router on the network that could do it, but it'll end up pushing a lot of data through. This is going to be used as part of a storage cluster. It'll be running ceph mon under a virtual machine while the ceph ods's will be on bare metal hardware connected via infiniband.
    – hookenz
    Sep 11, 2013 at 2:17
  • Routing seems like the best option for me at this time. It works but I just need to add routes to my virtual machines. Michael if you answer the question with that I'll accept it.
    – hookenz
    Sep 18, 2013 at 21:34
  • Another alternative is SR-IOV which I'm now looking into.
    – hookenz
    Sep 26, 2013 at 4:09

2 Answers 2

1

IPoIB doesn't work for L2 bridging, as it's L3 over IB encapsulation. You can use eIPoIB instead by Mellanox: * http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

It's a new feature in version 2.0-2.0.51 as specified in the Release Notes. More details on eIPoIB here: * https://www.openfabrics.org/ofa-documents/doc_download/510-ethernet-tunneling-over-ipoib.html

3
  • Theres also EoIP. What's the difference between eIPoIB and EoIP?
    – hookenz
    Sep 18, 2013 at 21:37
  • what is EoIP? are you referring to some kind of VXLAN/NVGRE running over IPoIB? (at the end, we're looking for Ethernet over X...Y over InfiniBand). The goal is to be able to run Ethernet traffic over InfiniBand network. eIPoIB translates IPoE traffic to IPoIB. so at the end of the day, the traffic on the wire is IPoIB-compliant [RFC 4391]. this allows the host to talk with any existing machine that has the standard IPoIB driver.
    – LIUB
    Sep 19, 2013 at 22:02
  • eIPoIB isn't available in the mainline kernel. It seems to have died. I've since discovered virtual ipoib which has been in the kernel since 3.7.
    – hookenz
    Nov 25, 2014 at 23:22
0

To answering my own question - yet again.

The best alternative aside from routing via Dom0 is to use SR-IOV. Basically this creates a number of virtual cards (Actually Virtual Functions) that you then pass through to a guest. The guest must load a suitable driver. However, from the guest perspective it thinks it's using the card natively as though it has it all to itself.

SR-IOV provides the best performance in this scenario.

Getting this working requires a recent kernel (i'm using 3.11.4) and a version of the subnet manager with alias GUID support i.e. opensm version >= 3.3.14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .