Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

0
votes
1answer
28 views

Setting up a Hive Remote Metastore with Kerberos

I'm new to hive and trying to set it up in a relatively secure manner for a test environment. I want to use a remote metastore so MR jobs can access the DB. I seem to have things almost working, but ...
0
votes
1answer
153 views

Setting up multiple Hadoop clusters [closed]

Hopefully this question isn't off topic. I asked this on AskUbuntu but haven't gotten a responses: I've followed Michael Noll's excellent guide to get Hadoop set up on a single cluster: ...
2
votes
2answers
1k views

Hadoop HDFS Backup & DR Strategy

We are preparing to implement our first Hadoop cluster. As such we are starting out small with a four node setup. (1 master node, and 3 worker nodes) Each node will have 6TB of storage. (6 x 1TB ...
3
votes
0answers
3k views

How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?

I have a 5 slave Hadoop cluster (using CDH4)---slaves are where DataNode and TaskNode run. Each slave has 4 partitions dedicated to HDFS storage. One of the slaves needed a reinstall and this caused ...
0
votes
1answer
26 views

Is Elastic Workload Re-provisioning still possible with DataStax Enterprise 3.1?

In the documentation of DataStax Enterprise (DSE) 2.0 there is a chapter about "Elastic Workload Re-provisioning" (http://www.datastax.com/docs/datastax_enterprise2.0/workload_reprovisioning) but it ...
1
vote
0answers
361 views

Use Amazon SNS to send nagios alerts

Is there any way by which I can send nagios alerts to Amazon SNS ? I have tried with following steps, but its giving me this error in the Nagios log file: Jul 12 11:38:23 ip-10-134-13-204 nagios3: ...
2
votes
1answer
242 views

Hadoop - What is the purpose of the /usr/sbin/ shell scripts?

I am installing Hadoop 1.1.2 on CentOS 6.4. I read all the Hadoop documentation at http://hadoop.apache.org/docs/stable/ After installing, I noticed there are many shell scripts at /usr/sbin/. But ...
-1
votes
2answers
1k views

how do I install pdsh on centos 6? [closed]

I'm following a tutorial on how to configure a centos machine to be a node in a hadoop cluster for HortonWorks. I'm doing this on a virtual machine on VirtualBox. Sadly, since I am a linux beginner, ...
1
vote
2answers
429 views

Hadoop - /usr/bin/hadoop: line 320: /usr/bin/java/bin/java: Not a directory

I am installing Hadoop on CentOS 6.4. Following these instructions http://hadoop.apache.org/docs/stable/single_node_setup.html wget ...
0
votes
1answer
46 views

How do I reconfigure the default port or uninstall the interactive Javascript console in HDInsight?

The Microsoft HDInsight Preview is available for download and install via the Web Platform Installer, which bundles Hadoop on Azure installer. As part of the process, it installs something that ...
0
votes
2answers
343 views

ext4 on faulty disks. How to avoid remount read-only?

The Problem: I'm in charge for a Hadoop cluster of 44 nodes. We have 1.5TB WD Green Drives with (quite unknown) the Load Cycle Count problem. These disks work fine but as they get older they show an ...
0
votes
1answer
391 views

Unable to read S3 file from interactive pig job flow

I'm unable to read a simple test file on S3 from an interactive pig job flow (hadoop, elastic map reduce), and I'm not sure why. I have two S3 buckets. Let's call them unmounted_bucket, and ...
0
votes
1answer
80 views

installing and configuring hadoop on ubuntu

I'm unable to locate the file hadoop-ec2-env.sh. I've downloaded and intalled hadoop_1.0.4-1_x86_64.deb from http://mirror.olnevhost.net/pub/apache/hadoop/common/stable/ This is the most recent ...
0
votes
1answer
245 views

setting up environment variables for aws hadoop ec2

I've been following this book: Hadoop in Action It gives a nice guide on how to start using ec2 with hadoop. One of the first things that it says is to download the command line tools here: ...
0
votes
1answer
91 views

Hadoop: Slave nodes are not starting

I am trying to setup a Pseudo Distributed Hadoop Cluster on my machine. Env Details : Host OS: Windows Guest OS: Ubuntu Vm's Created one master and one slave. I was able to run the hadoop ...
0
votes
1answer
4k views

How do do this in apt.conf - Acquire::http::Proxy “http://server:port/”;

Below are the instruction I have to follow for setting up cloudera. I am confused about the below on ubuntu 12.04. On Debian/Ubuntu systems, add the following property to /etc/apt/apt.conf: ...
9
votes
4answers
2k views

DIY Hadoop Cluster - Heat & Dust issues?

Following are links of my DIY 6-Node Hadoop Cluster using i3 Machines, What is the best possible way to protect my design from dust & provide better heat transfer? What should I use to cover ...
0
votes
1answer
60 views

Getting exposure to big data without having to set up the environment [closed]

Is anyone aware of any sandbox where an environment is already set up for big data processing? It can be hadoop, cassandra, pig etc... I'm a sql server programmer, and trying to get into big ...
0
votes
0answers
97 views

Cannot access data from hbase with amazone ec2

I have a single node Hadoop machine in which HBase is running in the Amazon EC2 instance. Due to some reason the server got restarted. So i need to start Hadoop and Hbase again. Then it's working ...
1
vote
1answer
151 views

MapReduce job is hung after 1 of 5 reducers completed on single-node environment

I have only one Data Node on my dev environment on EC2. I ran heavy MR job and in 6 hours noticed that 100% of mappers and 20% of reducers finished (1 of reducer shows 100% competition, other ones - ...
1
vote
1answer
242 views

Java process failure (hadoop, hbase)

Anytime I am running hadoop/hbase process from a command prompt I get an error: /usr/local/hadoop/bin/hadoop: line 320: /usr/lib/jvm/jdk1.7.0/bin/java: cannot execute binary file ...
1
vote
1answer
174 views

hadoop: port appears open locally but not remotelly

I am new to linux and hadoop and I am having the same issue as in this question. I think I understand what is causing it but I don't know how to solve it (Don't know what they mean by "Edit the ...
1
vote
1answer
155 views

Programmatically configure Hadoop cluster from python

Are there any python API's to configure a hadoop cluster (namenode, jobtracker, etc..)setup on an OpenStack cloud?? I have the IP addresss to the VM's and am looking for openstack api's to configure ...
0
votes
1answer
163 views

Hadoop hardware disk choice [closed]

We're evaluating the options of setting up a large hadoop cluster. Now we actually have the choice to choose from these 3 setups: 300x server with 12x 1TB disk 150x server with 12x 2TB disk 100x ...
1
vote
1answer
70 views

Data lost after Hdfs client was killed

I wrote a simple tool to upload logs to HDFS. And I found some curious phenomenon. If I run the tool in foreground and close it with "Ctrl - C", there will be some data in HDFS. If I run the tool in ...
1
vote
1answer
732 views

Hadoop hdfs namenode is throwing an error

Full list of error: hb@localhost:/etc/hadoop/conf$ sudo service hadoop-hdfs-namenode start * Starting Hadoop namenode: starting namenode, logging to ...
0
votes
2answers
2k views

How does hadoop decide what its nodes hostnames are?

Currently the urls generated by the jobtracker & namenode return either hostnames like bubbles.local or just bubbles. These end up not resolving unless the client machine has specified these in ...
1
vote
1answer
385 views

Implications of Multiple JobTracker nodes in a Hadoop cluster?

I get the impression that one can, potentially, have multiple JobTracker nodes configured to share the same set of MR (TaskTracker) nodes. I know that, conventionally, all the nodes in a Hadoop ...
2
votes
3answers
345 views

Hadoop ecosystem web dashboard

I am trying to find a tool, which would show me an overview of my Hadoop ecosystem - state, health, running tasks, etc. I tried to Google, but did not find any. Is there some nice useful tool?
3
votes
1answer
6k views

Can't connect to HDFS in pseudo-distributed mode

I followed the instructions here for installing hadoop in pseudo-distributed mode. However, I'm having trouble connecting to HDFS. When I execute this command : ./hadoop fs -ls / I get a ...
0
votes
2answers
1k views

Kernel panic - not syncing - Attempted to kill init

I am using Linux 2.6.32-33-server #70-Ubuntu 10.04.3 as Data-Nodes and Name-Node in my Hadoop cluster, but one of my data node is down since morning. When i restarted that particular system it ...
0
votes
1answer
70 views

Monitoring tools for non-distro Hadoop

I've built a Hadoop cluster by installing most packages manually (using binaries or source). I opted not to use a custom distribution like Cloudera, MapR or Hortonworks, since I wanted the flexibility ...
0
votes
1answer
948 views

Can't start Hadoop from an init.d script

I'm using CentOS 6.2. I'm trying to start Hadoop from an init.d script, but it's failing. This is what I see in boot.log : Retrigger failed udev events [ OK ] ...
2
votes
1answer
2k views

Deleting temp directory from HDFS

Is there a smart way of deleting old files from the hdfs /tmp directory? (Just to make sure, I am not talking about the unix FS /tmp)
0
votes
1answer
97 views

Hadoop Configuration Files - Who Needs What?

As I am setting up Hadoop, one question keeps popping in my mind but I can't find the answer. Which Hadoop configuration files need to be copied to which nodes. For example, I'm making changes to the ...
0
votes
1answer
139 views

Hadoop CDH4 Evaluation: Which Ubuntu would be preferred? Lucid or Precise [closed]

I'm setting up a CDH4 in AWS for evaluation (we already have a CHD3 running on Ubuntu Lucid) and I'd like advice regarding any known gotchyas that I'd be likely to encounter if running it on Lucid vs. ...
3
votes
1answer
563 views

Deploy Hadoop to Openstack

I'd like to deploy Hadoop to Openstack cloud. Is there any automatic way to do that? Anyone tried to do that? I'm looking for some devops like juju. I've never used juju and right now I'm going ...
1
vote
1answer
3k views

Raspberry Pi based Hadoop cluster [closed]

Is it at least possible to build Hadoop cluster from Raspberry Pi-based nodes? Can such a cluster meet hardware requirements of Hadoop? And if so, how much Raspberry Pi nodes are required to meet ...
1
vote
0answers
297 views

Hadoop commands are taking a very long time to return

I am logged in (via SSH) to the NameNode of my Hadoop cluster; the problem I am having is that any hadoop fs commands, even simple ones like hadoop fs -ls are completed quickly, but take many minutes ...
1
vote
2answers
2k views

Hadoop DataNode is giving me an incompatible namespace ID

When I run the start-all.sh script from my master node, some of my DataNodes fail to start; the log file reports a Java IOException: Incompatible Namespace IDs in /tmp/$MY_USER_NAME.
1
vote
0answers
81 views

Increase CDH4 diskspace

Initially i was using bundled CDH3 VM (Cloudera Version of Hadoop) later, i removed CDH3 and now I am using CDH4 on CentOS as a VM (through VMWare player) with host machine as Win7 64bit. I am in the ...
0
votes
1answer
169 views

How much power supply do I need for my server, and could a shortage be causing my odd crashing?

I have 5 servers, all with similar hardware (i7, four 2tb 7200rpm drives, two 4tb 5400rpm drives, 430 watt power supply), and lately the machines have been freezing up. This has gotten worse in the ...
1
vote
1answer
428 views

Hadoop initscript askes password

I have installed hadoop on my ubuntu 12.04 single node .I am trying to execute an init script to make the hadoop run on start up but it asks password every time i execute. #!/bin/sh ### BEGIN ...
0
votes
3answers
3k views

Hadoop slave nodes not connecting

I have been trying to set up a Hadoop cluster; I managed to get it running in pseudo-distributed mode, and my one machine wordcounted Tolstoy's War and Peace in about thirty seconds. I am now trying ...
1
vote
1answer
577 views

Hadoop on Ubuntu - two different install directories?

I recently installed Hadoop 1.0.3 from the .deb provided by Apache. The package installed correctly, but there seem to be two directories that have Hadoop-related files: /usr/share/hadoop has jars and ...
3
votes
2answers
285 views

PXE boot Linux - which directories must be writable?

I'm planning to set up a small Hadoop cluster where the slave nodes boot and run from a central PXE server, to simplify deployment and updates, and to enable all of the disks on the slaves to be ...
2
votes
1answer
277 views

Setting up hive with hadoop

I am trying to set up the hive. I am using this guide: https://cwiki.apache.org/Hive/gettingstarted.html and I'm stuck at setting up /tmp and /user/hive/warehouse dirs. First of all, it seems to me a ...
0
votes
1answer
718 views

Empty homedir name when SSHing to localhost

I am trying to start a system (Hadoop, but that should not matter much for this question), and need to be able to ssh to localhost. I do this on windows with cygwin. The cygwin SSHD service is ...
0
votes
1answer
332 views

Access HDFS HADOOP using APACHE Web Server, Linux CentOS

If I have an apache web server as a directory how I can access to HDFS Cluster for upload and modify files, what is the configuration i want to do? many thanks
1
vote
1answer
344 views

What version of HDFS is compatible with HBase stable?

HBase stable is currently hbase-0.90.4, what version(s) of HDFS is it compatible with?