Tag Info

Hot answers tagged

8

This is a large topic that gets complicated fast. The CAP theorem is a good starting point, as it identifies the higher level choices that must be made. When you are dealing with a write heavy Web application, it makes it more difficult to distribute load across the Internet while maintaining data integrity. Read centric applications (search!) are easier ...


7

What about Windows Distributed File System? That should allow you to create one drive that is in reality a mixture of multiple partitions across different disks/servers


7

Two options that come to mind are GlusterFS and Hadoop HDFS.


6

SPF is not a security credential I think the big misconception here is that SPF records provide a trusted credential. In truth SPF is not a secure credential, at least in the real world of security. DNS is an inherently insecure protocol. It's based on UDP (no transport level handshake), the protocol itself does not implement a handshake, and there is no ...


5

For services that need to always be available, you need N+1 redundancy, where N is the number of datacenters or servers (or whatever else you lose in the proposed failure scenario) needed to handle the load. This gets less expensive bigger you get - at the low end with two datacenters each needs to be able to handle the entire workload. But if you have 10, ...


5

Think about how this would work. SPF works because it allows remote mail servers to verify their incoming mail to ensure that the mail has in fact come from where it's meant to. This puts the onus on the sender to have the SPF record in the first place. How would this work with a DNS record for web access? It publishes a list of clients who are allowed to ...


4

A fairly common approach is that for the production environment the hard reserved capacity is sufficient that in case of calamity the remaining datacenter(s) ought to be able to handle the full load and all operations continue business as usual. Typically budgets never stretch far enough nor is the apparent business-case viable to allow full disaster ...


4

IBM's GPFS can do this (note: not open-source). With GPFS you can create Network-Shared-Disks (NSDs) that are comprised of any type of block-storage (local or presented via iSCSI or FC, for example). It would be entirely possible to create a GPFS filesystem (device) that is comprised of NSDs that span each 2TB hard drive across your 100 servers. I won't ...


4

If you want to contribute to humanity, look into Folding@Home or any of these distributed computing projects. Or, if you'd rather make a few cents for yourself, look into Bitcoin mining.


3

So basically, why should you do parallel processing over a network of machines when it can rather be done at much lower cost and much more reliably on 1 machine that supports numerous cores? You should do parallel processing over a network of machines when it can't be done at much lower cost and more reliability on one machine that support numerous ...


3

Cost: Thousand of cores on one server compared to several low-end x86 servers. Reliability: one server instead of many servers.


3

Having different servers in different locations shouldn't be a problem, until they can reach each other. The problem would be the bandwidth between them and what you make flow on it. Heartbeat doesn't use snmp and could be multicast, unicast or broadcast. It's a specific protocol (anyway snmp works between lans sicne it's a udp protocol). What kind of ...


3

I have not had a chance to play with it yet so I can't give a full review, but I would say have a look at the Openstack cloud stuffs -> http://www.openstack.org/projects/storage/


3

Corporately we use grid computing for things similar to this. A great one to investigate is Condor.


3

What you're probably looking for is called Single System Image. It's not a popular approach due to it own complexity and has it's own set of issues. For instance what would happen if a node went down or offline and took a running thread with it. So your program or an API would have to be able to detect and deal with that issue. So I don't think ...


3

Let me jsut say: cheaper to buy better boxes. No joke - 8 cores, 10g ram is a low end server. There ARE 2 providers of "combined VM technology" but they are commercial and it COSTS. Like it is needed of you neded t o combine high end boxes. Single core license costs more than your outdated desktops combined, sadly. So, the asnwer is: no way. Check MOSIX ...


2

What type of distributed/parallel computing infrastructure you build depends a lot on the problem being worked on. The easiest workloads to distribute are those that are easily subdivide-able: carve the problem-set into 4 chunks, farm the chunks to 4 machines, stitch the results back together once processing is done. Workloads that are poor choices for ...


2

Sounds like you're most of the way to what you need. The technologies out there (GlusterFS, GPFS) have the features you're looking for but not the data-locality one. Depending on what you're doing with the data, this could be built into your job dispatcher. To me it sounds like you need to build in an indexing stage to your processing pipeline that ...


2

are those 'cpu intensive things'... developed in-house? then you'll have to split and distribute the job yourself. there are some nice libraries, but everything is very low level. prebuilt software? then talk to the developer/provider. either it's supported or it's not same process applied to lots of individual items (for example, processing thousands ...


2

Plan 9 from Bell Labs? I asked a similar question on ServerFault recently: http://serverfault.com/questions/89269/is-there-a-way-to-do-something-like-lvm-over-nfs. Solutions suggested there were Lustre and GlusterFS.


2

You probably want something like PVFS.


2

There is a full list on Wikipedia.


2

This is very much dependent on the implementation at hand (both of the cluster and the actual computer job), which in turn is dependent on the type of problems it tries to solve. There are computing problems that are impossible to compute in parallel systems, while others have demand for extremely fast IPC or are independent of each other and scale quasi ...


2

Use a configuration management system, like Chef or Puppet. Have the configuration management server push out the appropriate cron/Upstart/monit/whatever configurations to the various nodes, depending on their roles. Yes, it's probably more of an investment to set this up than spewing crontab files all over the place manually, but you will wind up with a ...


2

In this regard, Amazon's EC2 instances are no different than any other hardware. Yes, there are frameworks that help with cluster computing, but it's not simple, and there is surely no "magic" package you can install to make it work. Your application will need to be written with parallelization/clustering in mind for this to have any chance of working, and ...


2

This totally depends on the workload you need to run on the system and how well it can scale in a clustered environment.


2

This is a very broad question and there is no silver bullet to solve this problem. The biggest challenge in setting up multiple sites is the Database, especially multi-master databases. Mysql and a bunch of nosql databases do support multi-master replication, you would need to evaluate and figure out which one fits your requirement the best. Slightly off ...


2

resource owner to restrict the access to one or more domain names. The closest thing to this I can think of would be to run all your own name servers, and use ACL's to restrict who is allowed to query a given zone. Example config excerpt (for bind): acl trusted_src_ip { 192.168.2.0/24; 2001:db8::/64; }; zone "mysecretdomain.com" IN { type ...


2

No, not with the HARDWARE you have. Not cost effectively, at least. Ther are some Hypervisors out there that allow you to combine multiple machines into one - but that is a fringe system and has STEEP hardware requirements. As in: You need a FAST network between them. FOrget 10gigabit - you talk of multiple 40g Infiniband links. Given the seriously low ...


1

It sounds like you've already done a lot of the groundwork here. With respect to your key requirements, all of the filesystems you identified meet them to some extent. These are things distributed filesystems do by virtue of being distributed filesystems. The only requirement that merits further discussion is licensing -- this is presumably a business ...



Only top voted, non community-wiki answers of a minimum length are eligible