Sign up ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

My task: I need to assemble computational cluster, consists of two nodes using infiniband technology.

When I try to start mpirun ... for two nodes, the process hangs. If i try to use flag -display-devel-map, it says that the daemons on other node not work.

There is configuration of my hardware and software: 2 x nodes:


  • Supermicro intel C602 chipset
  • CPU: 2 x Intel Xeon E5-2620
  • GPU: 4 x Nvidia Tesla M2090
  • Ethernet: Intel gigabit 2 ports.


  • Infiniband: 2 x Mellanox QDR infinihost
  • IB Switch: 1 x Mellanox switch 8-ports.
  • Operating system: RHEL 6.1 (without cluster suite).

What I did:

  • Install clear RHEL 6.1 (without load balancing and infiniband support)
  • Install Mellanox OFED ( from oficial site ), for dependencies i install "tcl" and "tk".
  • Set up the Ethernets interfaces - ping and ssh succes!
  • Set up the Infiniband interfaces - ping and ssh succes!
  • Set up rsa keys for passwordless acces. Test for both nodes!
  • Install OpenMPI ver 1.4 and 1.6. configured with:

    --prefix=/usr/local/mpi/openmpi16|14 --with-threads --with-hwloc \ --with-openib --enable-mpi-thread-multiple --with-mxm=/opt/mellanox/mxm \ --with-mxm-libdir=/opt/mellanox/mxm/lib --with-fca=/opt/mellanox/fca \ --enable-heterogeneous --enable-openib-connectx-xrc

But it says nothing to be done for this flags:

--with-openib --with-mxm=/opt/mellanox/mxm \ 
--with-mxm-libdir=/opt/mellanox/mxm/lib \
--with-fca=/opt/mellanox/fca  --enable-openib-connectx-xrc

When I try to mpirun only on localhost - all works correctly, both over IB or ETH - the mpi says about ib-buffer memory.

When I use mpirun -np 2 -hostfile nodes uname, where in nodes -names of both nodes, process hangs. I try to use hostnames and ip-addreses. This problem for both interfaces.

share|improve this question
Hi and welcome to Server Fault. Please do not cross post on the variosus Stack Exchange websites. This is discouraged and against the rules outlined in the FAQ. Cross post URL:… –  slm May 12 '13 at 6:06
Do you have opensm running? It may not be required if your switch does manangement. Orange link lights normally indicate an active connection. Next step would be to try ibping -S on one host and run ibping [other hostname] on the other machine to verify your network is working. In regards to OpenMPI, it will try to run with the fastest interconnect available. You can specify interconnect on the command line. –  chuck Nov 26 '13 at 14:48

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.