Tag Info

Hot answers tagged

8

This is a large topic that gets complicated fast. The CAP theorem is a good starting point, as it identifies the higher level choices that must be made. When you are dealing with a write heavy Web application, it makes it more difficult to distribute load across the Internet while maintaining data integrity. Read centric applications (search!) are easier ...


7

What about Windows Distributed File System? That should allow you to create one drive that is in reality a mixture of multiple partitions across different disks/servers


7

Two options that come to mind are GlusterFS and Hadoop HDFS.


4

IBM's GPFS can do this (note: not open-source). With GPFS you can create Network-Shared-Disks (NSDs) that are comprised of any type of block-storage (local or presented via iSCSI or FC, for example). It would be entirely possible to create a GPFS filesystem (device) that is comprised of NSDs that span each 2TB hard drive across your 100 servers. I won't ...


4

If you want to contribute to humanity, look into Folding@Home or any of these distributed computing projects. Or, if you'd rather make a few cents for yourself, look into Bitcoin mining.


3

Cost: Thousand of cores on one server compared to several low-end x86 servers. Reliability: one server instead of many servers.


3

So basically, why should you do parallel processing over a network of machines when it can rather be done at much lower cost and much more reliably on 1 machine that supports numerous cores? You should do parallel processing over a network of machines when it can't be done at much lower cost and more reliability on one machine that support numerous ...


3

Let me jsut say: cheaper to buy better boxes. No joke - 8 cores, 10g ram is a low end server. There ARE 2 providers of "combined VM technology" but they are commercial and it COSTS. Like it is needed of you neded t o combine high end boxes. Single core license costs more than your outdated desktops combined, sadly. So, the asnwer is: no way. Check MOSIX ...


3

What you're probably looking for is called Single System Image. It's not a popular approach due to it own complexity and has it's own set of issues. For instance what would happen if a node went down or offline and took a running thread with it. So your program or an API would have to be able to detect and deal with that issue. So I don't think ...


3

I have not had a chance to play with it yet so I can't give a full review, but I would say have a look at the Openstack cloud stuffs -> http://www.openstack.org/projects/storage/


3

Having different servers in different locations shouldn't be a problem, until they can reach each other. The problem would be the bandwidth between them and what you make flow on it. Heartbeat doesn't use snmp and could be multicast, unicast or broadcast. It's a specific protocol (anyway snmp works between lans sicne it's a udp protocol). What kind of ...


3

Corporately we use grid computing for things similar to this. A great one to investigate is Condor.


2

Plan 9 from Bell Labs? I asked a similar question on ServerFault recently: http://serverfault.com/questions/89269/is-there-a-way-to-do-something-like-lvm-over-nfs. Solutions suggested there were Lustre and GlusterFS.


2

are those 'cpu intensive things'... developed in-house? then you'll have to split and distribute the job yourself. there are some nice libraries, but everything is very low level. prebuilt software? then talk to the developer/provider. either it's supported or it's not same process applied to lots of individual items (for example, processing thousands ...


2

Sounds like you're most of the way to what you need. The technologies out there (GlusterFS, GPFS) have the features you're looking for but not the data-locality one. Depending on what you're doing with the data, this could be built into your job dispatcher. To me it sounds like you need to build in an indexing stage to your processing pipeline that ...


2

What type of distributed/parallel computing infrastructure you build depends a lot on the problem being worked on. The easiest workloads to distribute are those that are easily subdivide-able: carve the problem-set into 4 chunks, farm the chunks to 4 machines, stitch the results back together once processing is done. Workloads that are poor choices for ...


2

There is a full list on Wikipedia.


2

You probably want something like PVFS.


2

This is very much dependent on the implementation at hand (both of the cluster and the actual computer job), which in turn is dependent on the type of problems it tries to solve. There are computing problems that are impossible to compute in parallel systems, while others have demand for extremely fast IPC or are independent of each other and scale quasi ...


2

This is a very broad question and there is no silver bullet to solve this problem. The biggest challenge in setting up multiple sites is the Database, especially multi-master databases. Mysql and a bunch of nosql databases do support multi-master replication, you would need to evaluate and figure out which one fits your requirement the best. Slightly off ...


2

This totally depends on the workload you need to run on the system and how well it can scale in a clustered environment.


2

Use a configuration management system, like Chef or Puppet. Have the configuration management server push out the appropriate cron/Upstart/monit/whatever configurations to the various nodes, depending on their roles. Yes, it's probably more of an investment to set this up than spewing crontab files all over the place manually, but you will wind up with a ...


2

No, not with the HARDWARE you have. Not cost effectively, at least. Ther are some Hypervisors out there that allow you to combine multiple machines into one - but that is a fringe system and has STEEP hardware requirements. As in: You need a FAST network between them. FOrget 10gigabit - you talk of multiple 40g Infiniband links. Given the seriously low ...


1

easy way how to distribute my whole database Think about locking for a second. When 2 clients want to write to the same row in a database, the database uses write locks to avoid race conditions and invalid data. In a 'distributed database' scenario the acts of acquiring and releasing a lock themselves need to be distributed. How would you do that, how ...


1

The cluster gives you a little more power and some redundancy(if it's feasible for your software). The big 'workstation' has the advantage of being simpler to deploy and won't be bottlenecked by your switch. I can't say for certain if the switch will bottleneck since it will depend on your transfer sizes, etc.


1

Consider installing TORQUE. Its scheduler isn't the best out there, but it's more than sufficient for this kind of usage. You can replace the scheduler with Maui if you need the extra features later. The only feature from your list TORQUE misses is automatically retrying a job on failure. But you should be able to script that yourself on the TORQUE server ...


1

Ceph is another, though not prod ready yet. http://ceph.newdream.net/wiki/Main_Page


1

I use Thinkbox Deadline. It can notify you by email when render job is finished. You can also monitoring render job status in real time using mobile device like Android or iOS.


1

We use Sun Grid Engine and have just a few specified queues. A good example would be limiting half the cluster for long jobs, then have the cluster available for 1~2 hour jobs. This creates queues that people can get lots of work done on, and no one will be able to hog the whole thing.


1

What platforms and apps are you using? I can't speak for university setups in particular (I work for a VFX studio), but maybe this is helpful... if you have more specific questions I can probably answer them. drqueue is supposed to be pretty good for free/open source render queues. My current job uses Pixar's Alfred which is another option if you have ...



Only top voted, non community-wiki answers of a minimum length are eligible