Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have 5 PC(s) connected over a LAN through a switch.
I want to connect them to form a HPC cluster. The OS may be any Linux version (currently I have installed Ubuntu 8.10, 9.10 and Fedora 10)
Purpose of the Cluster
1. To execute my C code developed using OpenMP and provide high performance.
2. To act as load balancing cluster for my JSP dynamic web document I developed.

I would prefer to develop one cluster solving both the purposes, but in worst case would expect my cluster to atleast facilitate with the first purpose.

I would also prefer to develop a cluster which does not have a bottleneck of one node which if fails shall imply failure of the whole system. Instead, I want my cluster to continue working with remaining nodes even if few (even the most important) nodes fail...

Help of any kind shall be appreciated.

Thanks and Regards

share|improve this question
add comment

migrated from superuser.com Feb 17 '10 at 18:55

This question came from our site for computer enthusiasts and power users.

5 Answers

I don't think that you can do what you want with OpenMP. OpenMP is designed for shared memory programming. One example of this is using multiple cores and/or processors on the same machine. A cluster (at least the typical clusters made from commodity hardware) are distributed memory. Commonly MPI is used to program these types of machines.

There are several solutions out there to manage distributed memory clusters. The one that I'm familiar with is xcat and warewulf. Generally these types of systems have a job queue that runs jobs on free nodes. If a node fails, any job that is using that node will probably fail, but jobs will continue to be scheduled on the remaining nodes.

share|improve this answer
1  
Not true. You can run OpenMP code along side an MPI stack such as OpenMPI. See for example this mixed-mode MPI/OpenMP tutorial: cslab.ntua.gr/courses/pps/files/… –  Kamil Kisiel Feb 17 '10 at 21:15
1  
Sure, you can go for a hybrid solution. I interpreted the question as only using OpenMP, for which a cluster won't help unless you just want to run a bunch of single node OpenMP runs at the same time. –  KeithB Feb 17 '10 at 22:19
add comment

I would take a look at GridEngine (any of the variants). It's easy to install on Ubuntu and it has excellent support for running a variety of application utilizing the same resources. You can designate different queues for different types of applications and even subordinate queues so that higher priority tasks can bump lower priority tasks off the nodes.

In your situation you could submit load balancer tasks as jobs to the scheduler that could to some extent be pre-empted by your HPC tasks when necessary.

As far as resiliency to node failure, that is certainly covered. Generally unless you have a task that spans multiple nodes the failure of a single execution node should have no effect on the rest of your cluster.

share|improve this answer
    
Just for posterity: Nowadays, you should search for [Son of Grid Engine] (arc.liv.ac.uk/trac/SGE). –  jstarek Jul 26 '12 at 14:00
    
changed it to be more general. There are other forks (Open Grid Scheduler, Univa, even Oracle) that people might be interested in as well. –  Kamil Kisiel Jul 26 '12 at 20:50
add comment

If you just want to run simple applicattions and don't want your cluster running 24/7 You should use PelicanHPC to make a instant HPC. pareto.uab.es/mcreel/PelicanHPC Just boot PelicanHPC from the master node and boot the another computers trought network. Tutorial: http://pareto.uab.es/mcreel/PelicanHPC/Tutorial/PelicanTutorial.html

share|improve this answer
add comment

Setup a 'Beowulf' with a pretty good queuing system

share|improve this answer
add comment

Maybe you could go with Rocks Clusters, a distribution design to fit the needs of your first point. It is mainly used in scientific environments and is relatively easy to setup across several nodes (PXE). It also contains prepackaged software like several MPI implementations (OpenMPI, mpich, ...) and the Sun Grid Engine. The whole distribution is based on CentOS/RHEL, so if you are already familiar with Fedora you should not have any troubles during installation.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.