2

I have 5 PC(s) connected over a LAN through a switch.
I want to connect them to form a HPC cluster. The OS may be any Linux version (currently I have installed Ubuntu 8.10, 9.10 and Fedora 10)
Purpose of the Cluster
1. To execute my C code developed using OpenMP and provide high performance.
2. To act as load balancing cluster for my JSP dynamic web document I developed.

I would prefer to develop one cluster solving both the purposes, but in worst case would expect my cluster to atleast facilitate with the first purpose.

I would also prefer to develop a cluster which does not have a bottleneck of one node which if fails shall imply failure of the whole system. Instead, I want my cluster to continue working with remaining nodes even if few (even the most important) nodes fail...

Help of any kind shall be appreciated.

Thanks and Regards

Improve this question

5 Answers 5

Reset to default
2

I don't think that you can do what you want with OpenMP. OpenMP is designed for shared memory programming. One example of this is using multiple cores and/or processors on the same machine. A cluster (at least the typical clusters made from commodity hardware) are distributed memory. Commonly MPI is used to program these types of machines.

There are several solutions out there to manage distributed memory clusters. The one that I'm familiar with is xcat and warewulf. Generally these types of systems have a job queue that runs jobs on free nodes. If a node fails, any job that is using that node will probably fail, but jobs will continue to be scheduled on the remaining nodes.

Improve this answer
2
  • 1
    Not true. You can run OpenMP code along side an MPI stack such as OpenMPI. See for example this mixed-mode MPI/OpenMP tutorial: cslab.ntua.gr/courses/pps/files/…
    – Kamil Kisiel
    Feb 17, 2010 at 21:15
  • 1
    Sure, you can go for a hybrid solution. I interpreted the question as only using OpenMP, for which a cluster won't help unless you just want to run a bunch of single node OpenMP runs at the same time.
    – KeithB
    Feb 17, 2010 at 22:19
1

I would take a look at GridEngine (any of the variants). It's easy to install on Ubuntu and it has excellent support for running a variety of application utilizing the same resources. You can designate different queues for different types of applications and even subordinate queues so that higher priority tasks can bump lower priority tasks off the nodes.

In your situation you could submit load balancer tasks as jobs to the scheduler that could to some extent be pre-empted by your HPC tasks when necessary.

As far as resiliency to node failure, that is certainly covered. Generally unless you have a task that spans multiple nodes the failure of a single execution node should have no effect on the rest of your cluster.

Improve this answer
2
  • Just for posterity: Nowadays, you should search for [Son of Grid Engine] (arc.liv.ac.uk/trac/SGE).
    – jstarek
    Jul 26, 2012 at 14:00
  • changed it to be more general. There are other forks (Open Grid Scheduler, Univa, even Oracle) that people might be interested in as well.
    – Kamil Kisiel
    Jul 26, 2012 at 20:50
1

If you just want to run simple applicattions and don't want your cluster running 24/7 You should use PelicanHPC to make a instant HPC. pareto.uab.es/mcreel/PelicanHPC Just boot PelicanHPC from the master node and boot the another computers trought network. Tutorial: http://pareto.uab.es/mcreel/PelicanHPC/Tutorial/PelicanTutorial.html

Improve this answer
0

Maybe you could go with Rocks Clusters, a distribution design to fit the needs of your first point. It is mainly used in scientific environments and is relatively easy to setup across several nodes (PXE). It also contains prepackaged software like several MPI implementations (OpenMPI, mpich, ...) and the Sun Grid Engine. The whole distribution is based on CentOS/RHEL, so if you are already familiar with Fedora you should not have any troubles during installation.

Improve this answer
-1

Setup a 'Beowulf' with a pretty good queuing system

Improve this answer

You must log in to answer this question.