0

I'd like to build my own private computing cloud, or so-called grid.

I have access to IT classrooms with a bunch of fast computers, and I'd like to utilize that huge horsepower. (Or even just use the 3 small computers at home at the same time for the same task.)

Something like a virtual machine running distributed on all the CPU cores would be great. So that I'd have a single (virtual) OS running with ~40GB RAM, ~120GHz effective frequency and lots of storage space. I know that the network will slow down everything, but if this "resource merge" doesn't worth it because of slow network (although it's 1 Gb/s on every machine), something like BOINC, but smaller and easier would do it too. Something that distributes workunits for local nodes to crunch on.

I'd like it to be free, maybe open source, and to run on Windows and Linux nodes.

If there's no easy way, may I implement my own protocol dedicated to each computation (that could be anything from chess AI to raytracing, from fractal rendering to PI calculating) I'd like to do? Or use BOINC?

I have taken a look at Eucalyptus, but I think that would be quite an overshoot. Wouldn't BOINC be that too?

Does anyone have any good software to recommend me with which I could accomplish this?

Improve this question
1

2 Answers 2

Reset to default
0

I'd have a single (virtual) OS running with ~40GB RAM, ~120GHz effective frequency and lots of storage space

Doesn't exist without custom hardware or software. The reason it doesn't exist is that your 'fast' 1GbE links are slow as hell when you compare them to the interconnect speeds inside any given PC. Here's a few examples:

  • PCI - 2Gb/s
  • AGPx8 - 34Gb/s
  • PCIex8 - 16Gb/s
  • PC2-5300 RAM - 85Gb/s
  • CPU L1 cache ~300Gb/s

Now you can do a few of the individual things you mentioned separately. For example there are distributed computing applications like BOINC/SETI@Home, which if course rely on workloads that enjoy high parallelism (can be processed separately and combined later). There are also Distributed storage applications like Bittorrent/Brancecache.

If you have a particular workload in mind that you want to number crunch, and the programming chops, take a look at Beowulf clustering.

A really important concept to understand here is how the distance of components affects slower data transfer between them. This is a fundamental principle of computing and explains a lot of things, including the answer to your question. Short version - The further two components are from each other the slower their max communication speed. Consider how fast L1 cache is compared to RAM, then compared to the hard drive. This is a direct function of its distance from the core.

Improve this answer
1
  • Thank you for your very detailed answer. It was helpful. I understand that the network connections are usually (relatively) terribly slow, that's why I mentioned BOINC with its workunit distributing method. I think I will just write my own distributing server and protocol (and of course, computing client) specifically for every task I'd like to parallelize. With the basic idea same to BOINC's, but many times simpler and more primitive.
    – Attila
    Apr 10, 2011 at 19:11
0

This question is very broad. In general, running a single general-purpose OS over multiple computers is the holy grail of parallel/distributed computing, and has not been achieved in any meaningful way. If your problem domain is easily parallelizable, then there are many solutions for your problem, including batch scheduling systems like Condor and clustering approaches such as Eucalyptus, Hadoop, and Beowulf. If it's not easily parallelizable, then you have a lot of algorithmic thinking to do. So, for any given problem space,

  • Can you divide the work into many small parts?
  • Do the separate workers need to communicate with each other, or only with the dispatcher?
  • Do the separate workers need access to each others' memory spaces?
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .