7

My post below was (I think rather unfairly) closed on StackOverflow hence my reposting here.

Original Question: How can I use several computers to create a faster environment? I have about 12 computers with 4GB each and 2GHz each. I need to run some time consuming data transform and would like to use the combined power of these machines. They are all running Win2003 server.

Basically we have a large number of video files that we need to transform so our analysts can do their analysis. The problem is complicated by the fact I can't tell you more about the project.

Original Post: https://stackoverflow.com/questions/1126710/is-it-possible-to-create-a-faster-computer-from-many-computers

1

9 Answers 9

9

What you're asking about is at the crux of a major question in computing today. Individual processing cores aren't going to get much faster, so we need programmers to start writing code that breaks larger problems down into smaller problems that can be processed in parallel on multiple computers.

Short answer for you: If your processing software allows you to break jobs apart and run them in parallel already then do that. If it doesn't, then talk to the people who wrote it about having them re-tool it to work in a more parallel fashion.

The mechanics of getting the data out to the individual computers, starting up parallel tasks, making sure the jobs actually finished up, and bringing the data back is a lot of what the cluster management software that other posters are mentioning does. There are some non-trivial problems involved, but in general cluster management software is about job scheduling and resource management. The cluster management software doesn't handle actually doing the parallel work-- that's what your "processing software" is going to have to do.

There's no "magic" that you can throw at a group of multiple computers to make them "act like" a single faster machine. You're not going to get out of this w/o having software that's built to take advantage of multiple processors out of the gate.

5

To combine the processing power of multiple machines your going to want to run some sort of clustering software, often called a compute or computing cluster. Some examples of applicaitons that can do this are:

These are however complicated bits of software, creating a usable high performance cluster is a complicated and potentially expensive job and should not be undertaken lightly. Your also going to need special software that can run on a cluster to do your work. You can't simply connect a bunch of windows computers together and magically make a cluster, and install a standard bunch of Windows applications on it.

2

A cluster of computers works well when you can say...

Can I have someone sit at each computer and do part of the problem, and will that speed things up? In other words, can the problem be split into chunks that are basically independent?

Given that you describe the problem as "a large number of video files to transform", yes, this would work on a cluster.

Microsoft does have clustering software that I know nothing about. It might be the way to go.

Probably easier would be:

Dedicate one computer as the master. All the rest are clients.

Put all your video files on this master in a folder.

Make a shared folder per client and have each client mount that shared folder.

The clients run some windows scripting language (perl would work) which wakes up every so often, and, if something is in the shared folder, renames it to work_yourfilenamehere and runs your transform. When it's done it renames it to done_yourfilenamehere. If not, just wait for 60 secs or so.

The master computer runs a script which looks in each of the shared folders. If they have nothing in them then put one file to be transformed. If they have a file named done_yourfilenamehere then move it to the done folder. Loop until nothing is left in the master folder.

Basically all the clients should be kept as busy as possible, with each client taking what ever time they needed to transform each file.

0

It's possible, but there'd be a fair amount of work involved in building a cluster. It's difficult to say more without more information

3
  • Are you suggesting that "a custer" somehow can execute software that was otherwise written for a single computer "magically" in parallel w/o no modifications to the software? Have you ever actually used a cluster before? Jul 14, 2009 at 18:10
  • 1
    No, I'm suggesting that the question is hard to answer without more information. Jul 14, 2009 at 18:22
  • Fair enough. You were quick to the game here and I was getting a bit of a "magic cluster pixie dust" feel from your answer. My apologies for coming off like a jerk. Jul 14, 2009 at 19:36
0

The answer to your question depends largely on the software you are using to "transform" these video files, and what that entails.

Adobe After effects has a sort of clustering mode they call "network render" but it isn't an ideal solution. (last time I checked it involved rendering your video out to an imagine sequence, which would then presumably need to be recombined into a video for your use)

That basic idea is to split up the movie into chunks and get a different PC working on each chunk:

  • There should be some command-line video encoding tools that have options to only processing a certain range of frames; this could automate the splitting.

  • Each chunk could be copied to a different shared folder (again pretty easy to automate)

  • Each of the PC's in your "farm" would be looking at one of these shared folders for a source video to transform. (Very easy assuming the software doing the transformation has a function to "watch a folder", but a batch file / shell script could probably be rigged up for this)

Without knowing the nature of the video "transformation", what software you are using, or the level of scripting/programming at your disposal, it will be impossible to comment further.

0

what you're "creating" is not "a faster computer", but a cluster. which is nothing more than a group of computers used together.

IOW, what you have to do is to create some program that runs on each of your computers, and performs part of the problem.

In your specific case, you have many files to transform. if each file can be transformed independently of the others, it's almost easy: distribute them among the machines and put them all to transform their own files.

0

As others have said it depends what you're trying to do and how straightforward it is to execute in parallel.

Amdahl's law and Gustafson's law describe the amount of speed-up as the execution is distributed across more processors.

0

Have you considered Hadoop? Using Hadoop you can run Batch job like yours using multiple machines.

http://hadoop.apache.org/

-1

The majority of the biggest computers on the planet are large clusters of many individual nodes, each of which is often ver much a standard server.

See http://www.top500.org/, specifically the breakdown by architecture: http://www.top500.org/stats/list/33/archtype.

You must log in to answer this question.