Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have been enjoying reading ServerFault for a while and I have come across quite a few topics on Hadoop. I have had a little trouble finding out what it does from a global point of view.

So my question is quite simple : What is Hadoop ? What does it do ? What is it used for ? Why does it kick ass ?

Edit : If anyone happens to have demonstrations/explanations of use cases in which Hadoop was used, that would be fantastic.

share|improve this question

closed as not constructive by Bryan, Tom O'Connor, mdpc, growse, mgorven May 20 '13 at 18:47

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

Facebook makes heavy use of Hadoop (well really Hive which is a layer on top of Hadoop). There is a good writeup of it on the Facebook Engineering page. – John Meagher Jun 18 '09 at 17:30
Hadoop is a framework which makes the processing of large amount of data (Big data) simple by distributing the clusters of data among the nodes/servers and making the process run in parallel. This process/algorithm is known as MapReduce. – Mr_Green Jul 21 '14 at 6:03
up vote 24 down vote accepted

Straight from the horse's mouth:

Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or reexecuted on any node in the cluster. In addition, it provides a distributed file system (HDFS) that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both Map/Reduce and the distributed file system are designed so that node failures are automatically handled by the framework.

Map/Reduce is a programming paradigm that was made popular by Google where in a task is divided in to small portions and distributed to a large number of nodes for processing (map), and the results are then summarized in to the final answer (reduce). Google and Yahoo use this for their search engine technology, among other things.

Hadoop is a generic framework for implementing this kind of processing scheme. As for why it kicks ass, mostly because it provides neat features such as fault tolerance and lets you bring together pretty much any kind of hardware to do the processing. It also scales extremely well, provided your problem fits the paradigm.

You can read all about it on the website.

As for some examples, Paul gave a few, but here's a few more you could do that are not so web-centric:

  • Rendering a 3D film. The "map" step distributes the geometry for every frame to a different node, the nodes render it, and the rendered frames are recombined in the "reduce" step.
  • Computing the energy in a system in a molecular model. Each frame of a system trajectory is distributed to a node in the "map" step. The nodes compute the the energy for each frame,
    and then the results are summarized in the "reduce" step.

Essentially the model works very well for a problem that can be broken down in to similar discrete computations that are completely independent, and can be recombined to produce a final result.

share|improve this answer
Thank you for your answer. So basically it takes apps (PHP ? Java ?) and it breaks them down and dispatches the work amongst a bunch of nodes ? As for HDFS, it's sort of like OCFS except with a bunch of nodes ? – Antoine Benkemoun Jun 18 '09 at 6:47
Interested in this aswell. I'd like to see some more specific, real word examples though. – Karolis T. Jun 18 '09 at 7:19
That was what I was looking for also :-) – Antoine Benkemoun Jun 18 '09 at 7:20

Cloudera have some great videos that explain the principles behind Map Reduce and Hadoop.

One of the core ideas behind MapReduce is that for large data sets you are going to be io bound on your disks, so in Hadoop HDFS gives you the ability to split things up between lots of nodes enabling parallel processing.

Some uses of Hadoop of interest to systems administrators are often around processing large log file sets - I can only post one link but these include, google should find these:

  1. Rackspace mail log query
  2. Apache log analysis with pig - see Cloudera blog
  3. Yahoo! fight spam
share|improve this answer
Thanks for the link! – rkthkr Jun 18 '09 at 8:11
Looks nice I'll have a look :-) – Antoine Benkemoun Jun 18 '09 at 8:56

Initially hadoop is developed for large amount of data sets in OLAP environment.

With introduction of Hbase on top of hadoop, cane be used for OLAP Processing also. Hadoop is a framework with all the subcomponents like map reduce,hdfs,hbase,pig.

Ifound one the the article with basic of hadoop in Why Hadoop is introduced.

In Hadoop,data storage in the form of files, not in the tables,columns.

share|improve this answer

protected by Michael Hampton May 17 '13 at 21:27

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.