Tag Info

Hot answers tagged

8

This is a large topic that gets complicated fast. The CAP theorem is a good starting point, as it identifies the higher level choices that must be made. When you are dealing with a write heavy Web application, it makes it more difficult to distribute load across the Internet while maintaining data integrity. Read centric applications (search!) are easier ...


7

What about Windows Distributed File System? That should allow you to create one drive that is in reality a mixture of multiple partitions across different disks/servers


7

Two options that come to mind are GlusterFS and Hadoop HDFS.


4

IBM's GPFS can do this (note: not open-source). With GPFS you can create Network-Shared-Disks (NSDs) that are comprised of any type of block-storage (local or presented via iSCSI or FC, for example). It would be entirely possible to create a GPFS filesystem (device) that is comprised of NSDs that span each 2TB hard drive across your 100 servers. I won't ...


4

If you want to contribute to humanity, look into Folding@Home or any of these distributed computing projects. Or, if you'd rather make a few cents for yourself, look into Bitcoin mining.


3

So basically, why should you do parallel processing over a network of machines when it can rather be done at much lower cost and much more reliably on 1 machine that supports numerous cores? You should do parallel processing over a network of machines when it can't be done at much lower cost and more reliability on one machine that support numerous ...


3

Cost: Thousand of cores on one server compared to several low-end x86 servers. Reliability: one server instead of many servers.


3

Having different servers in different locations shouldn't be a problem, until they can reach each other. The problem would be the bandwidth between them and what you make flow on it. Heartbeat doesn't use snmp and could be multicast, unicast or broadcast. It's a specific protocol (anyway snmp works between lans sicne it's a udp protocol). What kind of ...


3

I have not had a chance to play with it yet so I can't give a full review, but I would say have a look at the Openstack cloud stuffs -> http://www.openstack.org/projects/storage/


3

Corporately we use grid computing for things similar to this. A great one to investigate is Condor.


3

What you're probably looking for is called Single System Image. It's not a popular approach due to it own complexity and has it's own set of issues. For instance what would happen if a node went down or offline and took a running thread with it. So your program or an API would have to be able to detect and deal with that issue. So I don't think ...


3

Let me jsut say: cheaper to buy better boxes. No joke - 8 cores, 10g ram is a low end server. There ARE 2 providers of "combined VM technology" but they are commercial and it COSTS. Like it is needed of you neded t o combine high end boxes. Single core license costs more than your outdated desktops combined, sadly. So, the asnwer is: no way. Check MOSIX ...


2

What type of distributed/parallel computing infrastructure you build depends a lot on the problem being worked on. The easiest workloads to distribute are those that are easily subdivide-able: carve the problem-set into 4 chunks, farm the chunks to 4 machines, stitch the results back together once processing is done. Workloads that are poor choices for ...


2

Sounds like you're most of the way to what you need. The technologies out there (GlusterFS, GPFS) have the features you're looking for but not the data-locality one. Depending on what you're doing with the data, this could be built into your job dispatcher. To me it sounds like you need to build in an indexing stage to your processing pipeline that ...


2

are those 'cpu intensive things'... developed in-house? then you'll have to split and distribute the job yourself. there are some nice libraries, but everything is very low level. prebuilt software? then talk to the developer/provider. either it's supported or it's not same process applied to lots of individual items (for example, processing thousands ...


2

Plan 9 from Bell Labs? I asked a similar question on ServerFault recently: http://serverfault.com/questions/89269/is-there-a-way-to-do-something-like-lvm-over-nfs. Solutions suggested there were Lustre and GlusterFS.


2

You probably want something like PVFS.


2

There is a full list on Wikipedia.


2

This is very much dependent on the implementation at hand (both of the cluster and the actual computer job), which in turn is dependent on the type of problems it tries to solve. There are computing problems that are impossible to compute in parallel systems, while others have demand for extremely fast IPC or are independent of each other and scale quasi ...


2

Use a configuration management system, like Chef or Puppet. Have the configuration management server push out the appropriate cron/Upstart/monit/whatever configurations to the various nodes, depending on their roles. Yes, it's probably more of an investment to set this up than spewing crontab files all over the place manually, but you will wind up with a ...


2

This totally depends on the workload you need to run on the system and how well it can scale in a clustered environment.


2

This is a very broad question and there is no silver bullet to solve this problem. The biggest challenge in setting up multiple sites is the Database, especially multi-master databases. Mysql and a bunch of nosql databases do support multi-master replication, you would need to evaluate and figure out which one fits your requirement the best. Slightly off ...


2

No, not with the HARDWARE you have. Not cost effectively, at least. Ther are some Hypervisors out there that allow you to combine multiple machines into one - but that is a fringe system and has STEEP hardware requirements. As in: You need a FAST network between them. FOrget 10gigabit - you talk of multiple 40g Infiniband links. Given the seriously low ...


1

It sounds like you've already done a lot of the groundwork here. With respect to your key requirements, all of the filesystems you identified meet them to some extent. These are things distributed filesystems do by virtue of being distributed filesystems. The only requirement that merits further discussion is licensing -- this is presumably a business ...


1

easy way how to distribute my whole database Think about locking for a second. When 2 clients want to write to the same row in a database, the database uses write locks to avoid race conditions and invalid data. In a 'distributed database' scenario the acts of acquiring and releasing a lock themselves need to be distributed. How would you do that, how ...


1

There are some advantages and disadvantages to EBS and instance store. In general, I recommend instance store over EBS for most scalable applications. Some advantages of EBS: Ability to stop/start instances Root volume size larger than 10 GB Ease of creating AMIs Ability to snapshot root volume Some advantages of instance store: Not dependent on ...


1

In this regard, Amazon's EC2 instances are no different than any other hardware. Yes, there are frameworks that help with cluster computing, but it's not simple, and there is surely no "magic" package you can install to make it work. Your application will need to be written with parallelization/clustering in mind for this to have any chance of working, and ...


1

The cluster gives you a little more power and some redundancy(if it's feasible for your software). The big 'workstation' has the advantage of being simpler to deploy and won't be bottlenecked by your switch. I can't say for certain if the switch will bottleneck since it will depend on your transfer sizes, etc.


1

Ceph is another, though not prod ready yet. http://ceph.newdream.net/wiki/Main_Page


1

Consider installing TORQUE. Its scheduler isn't the best out there, but it's more than sufficient for this kind of usage. You can replace the scheduler with Maui if you need the extra features later. The only feature from your list TORQUE misses is automatically retrying a job on failure. But you should be able to script that yourself on the TORQUE server ...



Only top voted, non community-wiki answers of a minimum length are eligible