I have two networks.
One is a standard Ethernet network running IP. The second network is Infiniband which in addition to some custom protocols can talks IPoIB (IP over Infiniband).
The router that sits on this network can ping both the Infiniband hosts with 10.10.10.x ip addresses, and can also ping local xx.xxx.79.x addresses.
The problem I'm having is that machines sitting on the ethernet network cannot ping or access machines on the infiniband network despite the fact that the router has ipv4 forwarding turned on.
Can you forward packets to/from IP and IPoIB Infiniband networks?
As requested, here is my routing table.
Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 xx.xxx.66.61 0.0.0.0 UG 100 0 0 eth2 10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0 xx.xxx.66.60 0.0.0.0 255.255.255.252 U 0 0 0 eth2 xx.xxx.79.128 0.0.0.0 255.255.255.128 U 0 0 0 bond0 255.255.255.255 0.0.0.0 255.255.255.255 UH 0 0 0 bond0
eth2 is the public internet, bond0 is a private LAN, ib0 is the infiniband network. This machine is the default router for both networks.
Note: this link seems to suggest it's possible. http://www.spinics.net/lists/linux-rdma/msg06784.html Although I've subsequently set this up on a test machine with old DDR hardware and am still seeing it not route. You also cannot bridge ib0 devices
Update and answer.
The problem I was having was actually related to routing. So yes it does work. In my particular setup, some of the hosts had infiniband and ethernet cards on the same network. So the replies were being passed back to the client ping via a different route. The answer in Linux is to set up reverse path filtering.
net.ipv4.conf.default.rp_filter = 2 net.ipv4.conf.all.rp_filter = 2
Then it works.