Forgive me if I use any of these terms incorrectly.

I am wondering if there is any kind of software that would allow my two "join" two computers together such that a cluster unaware application could utilize their combined computing resources? By "cluster unaware" I mean an application that isn't designed to share work across multiple services. My understanding is that clustering is enabled by the specific application by it's architecture, such that messaging with multiple instances of the application coordinate the sharing of work. Instead I am looking for something that enables clustering at the OS or virtualization level, so that any application could essentially be clustered.

Failing that, I am also wondering about the following scenario: We have 3 different applications we will call A, B, and C. We have 2 single core computers. At any given time lets say that any combination of those applications will be CPU intensive. In cases where only 2 of those apps are very active, have one of them moved over to a different server. In a nutshell, some sort of dynamic automatic shuffling of the application's load.

I have heard of virtual machines that can be migrated across physical machines while live, but I am wondering if this can be done automatically in response to an application's or VM's CPU activity?

  • Do you need a cluster for higher performance or for higher reliability? – chris Aug 6 '09 at 12:17
  • Not for reliability. More performance. I have many applications, but very little hardware. I don't really need more hardware because it would be rare that many of the applications will be active. Most of the time they will be idle, and at any given time maybe 2 or 3 will be active. I'd like to be able to have applications/VMs automatically shuffled between computers so that they aren't all grappling for a single machine. – AaronLS Aug 6 '09 at 12:24

You can virtualize the whole OS image and run your apps in these different virtualized OSes and move those OSes around to suite your loads. This is the ESX / Virtualization model. It can probably be traced back to IBM mainframe OSes where they virtualized the machine instead of making an operating system that could run several programs at once.

Hardware to hardware migrations are not super fast and you're going to burn lots of overhead on things like OS memory on each virtual instance.

Another option is to use something that presents a single OS image to the client programs but actually runs on lots of computers at once. An example of this model is mosix. Mosix extends linux such that a running program may under some situations be moved from one piece of hardware to another. The program itself doesn't realize this (just as it typically doesn't care if it moves from one CPU to another in a normal computer with 2 CPUs). There will be some overhead with this but it may be worth looking into.

  • mosix sounds like what I am looking for. Thanks. (I don't have enough rep to upvote you for now) – AaronLS Aug 6 '09 at 12:28
  • 1
    Just a little note for anyone else that might find it useful, this seems to be referred to as "single-system image (SSI) clustering". – AaronLS Aug 6 '09 at 12:31

If you install every app on a different VM, then yes, it can be live migrated using any given enterprise level VM technology

clustering for non clusterable software is possible if you're looking for failover clusters. Either use a VM and if the host dies, migrate the VM away to another host, or use software like XOsoft WANsync, NeverFail, DoubleTake etc.

resource sharing is a bit trickier, and the software generally has to be cluster aware, more so - it should be able to work in load balancing clusters, not just failover.

  • Regarding first sentence, do you know if there is any that have automatic live migration in response to CPU load? – AaronLS Aug 6 '09 at 12:12
  • AaronLS: I know how to set it up using RHEV – dyasny Aug 6 '09 at 15:33

Since MOSIX seems to be associated with MPI, it seems appropriate to mention there was a competing technology called PVM (Parallel Virtual Machine). It may be useful to explore if there is any MOSIX-like clustering OS designed based on PVM as well. I had used PVM to run C++ Tasks in parallel using multiple Linux servers as a single clustered HW. For example, running SQL Queries on humungous RDBMS rows could be partitioned across.

Besides providing a message passing interface, PVM makes network communication transparent, and supports flexible parallel computing in heterogeneous cluster systems.

In case you are using Myrinet switches HW, an optimized version of PVM can be used; see http://www.huaxiaspace.net/research/wed930.pdf.

I have seen some slides implying Mosix can be used with either MPI or PVM technology, but not sure.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.