0

I have been advised to learn Hadoop for my company and so I want to build a test environment.

The company I work for has a datacentre with a VMware vCloud infrastructure which they use to sell private clouds to other companies.

I'm allowed to use some CPU, RAM, network and storage, but I can't seem to find a base level requirements for Hadoop in a small testing environment.

I am thinking to spin up a single VM working as a single node cluster, working on simple job like analysing 1000s of PDF documents for keywords, extracting paragraphs and reformatting into a single HTML document.

Can anyone advise on the minimum requirements I should provision for this VM, or where I can find this information.

Thanks.

1 Answer 1

1

If you're just looking to learn how Hadoop works, I would recommend grabbing one of the available distributions on a VM. These are designed to be small learning environments that get you running with a pseudo-distributed cluster running inside one node.

  1. Cloudera Quickstart VM
  2. Hortonworks Sandbox
  3. MapR Virtual Machine

There may be others; these are just the ones I can think of off the top of my head.

If you're looking to build your own VM, the answer is: it depends.

You would probably be fine starting out with 8-16GB of RAM, a few hundred gigabytes of disk space, and 2-4 cpu cores. This would be sufficient to get you working in a pseudo-distributed configuration.

If you're talking about building a small cluster, you probably want 4 nodes: 1 namenode (2-4G memory, 1-2 CPUs), 3 datanodes (4-8GB of memory, as much space as you want/need). This would be sufficient to get you a running HDFS and Mapreduce environment AND give you experience on setting up and running the cluster. If you go this route, you will probably want to check out Cloudera Manager or Ambari with Hortonworks.

EDIT:

I almost forgot: vmware Serengeti may be of interest to you since you're in a VMware infrastructure.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .