1

We currently have a 2-node Oracle 11g RAC using 1GigE as the private interconnect. At peak update time, the traffic between nodes reaches 35 MByte/second, so we are still doing good in terms of bandwidth capacity. We are considering adding a third node to the cluster to support our growth in transaction volume. I am wondering whether somebody out there has already seen how adding nodes to the cluster would impact private interconnect traffic and what would trigger the use of Infiniband. From what i read, Infiniband not only can increase bandwidth but also has much shorter network latency. Can anybody who use Infiniband for RAC private interconnect share your experience with us? Thanks in advance.

Improve this question

4 Answers 4

Reset to default
2

I've a 4 node oltp rac database and a 3 node dw rac database using IB as interconnects.
We use cisco IB switches and cisco IB cards. We use IP over IB as the interconnect protocol. So far we are quite please with the results. Never had an issue with the interconnect traffic. We've come close to saturating our gigE public interface but no where close to the IB bandwidth. All 7 nodes are connected to 2 infiniband switches setup in active/passive failover configuration.

Improve this answer
1
  • Unfortunately, it appears that Cisco has completely abandoned IB in favor of 10GbE. Neither our Cisco reps nor our normal Cisco VAR are able to point to any Cisco Infiniband hardware for us to purchse.
    – Greeblesnort
    Aug 31, 2009 at 21:27
1

I've never used Infiniband in production, but as far as I can tell the only time it might make sense to go to Infiniband is if you find yourself erring towards 10GigE (if I remember price points kind of reach an equilibrium at about 10GbE). Latency measured in nanoseconds is pretty cool, but is it important for DB? Not really.

You can probably team two 1Gb connections (with appropriate hardware of course), or do any number of other tweaks (like Jumbo Packets, or adjusting MTU) before the need for something like Infiniband.

That said, Infiniband is very cool if you just want to brag to your mates...

Improve this answer
1

The big difference between IB and 10GbE is supposed to be latency. IB uses a DMA type model with latency in the microsecond range, while 10GbE requires your communication to travel up and back down the stack on both sides. IB is also supposed to have a higher raw bandwidth than 10GbE, but 10GbE will probably be backwards compatible when 40 & 100GbE come out.

We've actually just been tasked with building a 5 node Oracle cluster with IB interconnects and a IB -> 10GbE switch proof-of-concept that we're working with Dell on. So, my information is largely the research I've been doing in preparation for the POC.

Improve this answer
1

If you're more concerned about the cost (to your application) of bulk data transfers for instance-to-instance synchronization, then your current rate of 35Mbps is presently low enough that its very unlikely to be affected positively by moving to a different cluster interconnect.

Don't forget that you can very easily trunk your 1Gbps Ethernet connections to 4Gbps or 8Gbps (depending on the number of physical ports you can stick in your individual cluster nodes) without taking on more latency in the network stack.

As Farseeker says, IB is (in the commercial environment) cool to brag to your mates. In the scientific and research community, IB has almost become a commodity for cluster interconnect.

Improve this answer

You must log in to answer this question.