Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm going to build a high-performance web service. It should use a database (or any other storage system), some processing language (either scripting or not), and a web-server daemon. The system should be distributed to a large amount of servers so the service runs fast and reliable.

It should replicate data to achieve reliability and at the same time it must provide distributed computing features in order to process large amounts of data (primarily, queries on large databases that won't survive being executed on a single server with a suitable level of responsiveness). Caching techniques are out of the subject.

Which cluster/cloud solutions I should take for the consideration?

There are plenty of Single-System-Image (SSI), clustering file systems (can be a part of the design), projects like Hadoop, BigTable clones, and many others. Each has its pros and cons, and "about" page always says the solution is great :) If you've tried to deploy something that addresses the subject - share your experience!

UPD: It's not a file hosting and not a game, but something rather interactive. You can take ServerFault as an example of a web-service: small pieces of data, semi-static content, intensive database operations.


For those who might be interested:

Cross-Post on StackOverflow

Related questions:

share|improve this question

6 Answers 6

up vote 2 down vote accepted
+150

Facebook is using cassandra for data storage.

Here is article about scaling youtube and google architecture and prestentation: Designs, Lessons and Advice from Building Large Distributed Systems by Jeff Dean of Google describing how they do their thing.

share|improve this answer

Hadoop + Hive (or PIG) is built for dealing with massive data. This is what Yahoo (4000 node cluster), Facebook, eHarmony etc use.

I believe you can get branded packages/support from Cloudera.com, or you can get it yourself at apache.org

It is wicked easy to setup and it is awesome when dealing with GB-PB of data queries.

You could easily test it out on EC2 (that is one of their options) for almost no cost.

share|improve this answer
1  
Hadoop + Hive is for querying warehoused data ( wiki.apache.org/hadoop/Hive ). Distributed data stores like Cassandra ( incubator.apache.org/cassandra ), Voldemort ( project-voldemort.com ), and MongoDB ( mongodb.org ) are more tailored for low latency access to recent data that HPWS needs. –  Suppressingfire Nov 17 '09 at 15:48

It's impossible to answer without knowing exactly what you're doing; it may be rather difficult even then.

Based on what I've read (and tried out), Cassandra seems pretty good, but you should not consider it as part of a design without understanding exactly how it works and what its limitations are.

This kind of thing is never easy, and moreover, this is more of a question for Stackoverflow.

share|improve this answer
    
Thanks, I've duped it there. Added an example of something alike with the thing I need :) –  kolypto Nov 17 '09 at 15:15

I somewhat freely take the gist of OPs request to be "a mature cloud computing platform that is easy to grok for programmers, and easy to scale for operations". We are not quite there yet; to the best of my knowledge there are no mature, commercially available systems which span the entire chain from HTTP request, over processing, to permanent storage.

The closest thing today is probably partitioned data grid middleware like Oracle's Coherence or maybe Terracotta. Oracle Coherence has been good for Squarespace and other web applications. Of course Oracle would also happily sell you a partitioned Oracle database that can handle massive amounts of data and just works. And the price ... if you need to ask you can't afford it.

If you need cheap(er), then you're looking at some degree of do-it-yourself using open source components. The Hadoop family is the most comprehensive and mature open-source "BigTable" and "Map/Reduce" like set of tools. Sharded MySQL is popular for data storage, and is still a mostly DIY solution. "NoSQL" is gaining momentum right now, but it's still early days for NoSQL.

Which cluster/cloud solutions I should take for the consideration?

Don't you have it backwards? What evidence do you have of your application reaching Internet scale, what are the observed data access patterns at current scale like, and which solutions & languages does your team have prior experience with?

share|improve this answer
    
A very nice answer, thank you! Frankly, this question is a theoretical one rather than applied to any real application. In order to get acquainted with clustering I need some hints from people who designed/deployed these systems and have such experience :) // Languages are not really important: it can be C, Java, PHP, Pyton or whatever: when performance is the ultimate goal, any solution can be used. –  kolypto Nov 17 '09 at 19:43

I really like the full amazon suite.

I believe Amazon EC2 combined with Amazon S3 and Amazon Simple DB or Amazon RDS might be a good option for you

share|improve this answer

You could use a JBoss cluster.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.