Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

1
vote
1answer
8k views

How do do this in apt.conf - Acquire::http::Proxy “http://server:port/”;

Below are the instruction I have to follow for setting up cloudera. I am confused about the below on ubuntu 12.04. On Debian/Ubuntu systems, add the following property to /etc/apt/apt.conf: ...
9
votes
4answers
3k views

DIY Hadoop Cluster - Heat & Dust issues?

Following are links of my DIY 6-Node Hadoop Cluster using i3 Machines, What is the best possible way to protect my design from dust & provide better heat transfer? What should I use to cover ...
0
votes
1answer
63 views

Getting exposure to big data without having to set up the environment [closed]

Is anyone aware of any sandbox where an environment is already set up for big data processing? It can be hadoop, cassandra, pig etc... I'm a sql server programmer, and trying to get into big ...
1
vote
1answer
295 views

MapReduce job is hung after 1 of 5 reducers completed on single-node environment

I have only one Data Node on my dev environment on EC2. I ran heavy MR job and in 6 hours noticed that 100% of mappers and 20% of reducers finished (1 of reducer shows 100% competition, other ones - ...
1
vote
1answer
294 views

Java process failure (hadoop, hbase)

Anytime I am running hadoop/hbase process from a command prompt I get an error: /usr/local/hadoop/bin/hadoop: line 320: /usr/lib/jvm/jdk1.7.0/bin/java: cannot execute binary file ...
1
vote
1answer
237 views

hadoop: port appears open locally but not remotelly

I am new to linux and hadoop and I am having the same issue as in this question. I think I understand what is causing it but I don't know how to solve it (Don't know what they mean by "Edit the ...
1
vote
1answer
191 views

Programmatically configure Hadoop cluster from python

Are there any python API's to configure a hadoop cluster (namenode, jobtracker, etc..)setup on an OpenStack cloud?? I have the IP addresss to the VM's and am looking for openstack api's to configure ...
0
votes
1answer
202 views

Hadoop hardware disk choice [closed]

We're evaluating the options of setting up a large hadoop cluster. Now we actually have the choice to choose from these 3 setups: 300x server with 12x 1TB disk 150x server with 12x 2TB disk 100x ...
1
vote
1answer
107 views

Data lost after Hdfs client was killed

I wrote a simple tool to upload logs to HDFS. And I found some curious phenomenon. If I run the tool in foreground and close it with "Ctrl - C", there will be some data in HDFS. If I run the tool in ...
1
vote
1answer
1k views

Hadoop hdfs namenode is throwing an error

Full list of error: hb@localhost:/etc/hadoop/conf$ sudo service hadoop-hdfs-namenode start * Starting Hadoop namenode: starting namenode, logging to ...
1
vote
2answers
4k views

How does hadoop decide what its nodes hostnames are?

Currently the urls generated by the jobtracker & namenode return either hostnames like bubbles.local or just bubbles. These end up not resolving unless the client machine has specified these in ...
1
vote
1answer
620 views

Implications of Multiple JobTracker nodes in a Hadoop cluster?

I get the impression that one can, potentially, have multiple JobTracker nodes configured to share the same set of MR (TaskTracker) nodes. I know that, conventionally, all the nodes in a Hadoop ...
3
votes
3answers
838 views

Hadoop ecosystem web dashboard

I am trying to find a tool, which would show me an overview of my Hadoop ecosystem - state, health, running tasks, etc. I tried to Google, but did not find any. Is there some nice useful tool?
4
votes
1answer
10k views

Can't connect to HDFS in pseudo-distributed mode

I followed the instructions here for installing hadoop in pseudo-distributed mode. However, I'm having trouble connecting to HDFS. When I execute this command : ./hadoop fs -ls / I get a ...
0
votes
2answers
2k views

Kernel panic - not syncing - Attempted to kill init

I am using Linux 2.6.32-33-server #70-Ubuntu 10.04.3 as Data-Nodes and Name-Node in my Hadoop cluster, but one of my data node is down since morning. When i restarted that particular system it ...
0
votes
1answer
75 views

Monitoring tools for non-distro Hadoop

I've built a Hadoop cluster by installing most packages manually (using binaries or source). I opted not to use a custom distribution like Cloudera, MapR or Hortonworks, since I wanted the flexibility ...
0
votes
1answer
2k views

Can't start Hadoop from an init.d script

I'm using CentOS 6.2. I'm trying to start Hadoop from an init.d script, but it's failing. This is what I see in boot.log : Retrigger failed udev events [ OK ] ...
2
votes
2answers
3k views

Deleting temp directory from HDFS

Is there a smart way of deleting old files from the hdfs /tmp directory? (Just to make sure, I am not talking about the unix FS /tmp)
0
votes
1answer
121 views

Hadoop Configuration Files - Who Needs What?

As I am setting up Hadoop, one question keeps popping in my mind but I can't find the answer. Which Hadoop configuration files need to be copied to which nodes. For example, I'm making changes to the ...
0
votes
1answer
145 views

Hadoop CDH4 Evaluation: Which Ubuntu would be preferred? Lucid or Precise [closed]

I'm setting up a CDH4 in AWS for evaluation (we already have a CHD3 running on Ubuntu Lucid) and I'd like advice regarding any known gotchyas that I'd be likely to encounter if running it on Lucid vs. ...
2
votes
1answer
595 views

Deploy Hadoop to Openstack

I'd like to deploy Hadoop to Openstack cloud. Is there any automatic way to do that? Anyone tried to do that? I'm looking for some devops like juju. I've never used juju and right now I'm going ...
1
vote
1answer
3k views

Raspberry Pi based Hadoop cluster [closed]

Is it at least possible to build Hadoop cluster from Raspberry Pi-based nodes? Can such a cluster meet hardware requirements of Hadoop? And if so, how much Raspberry Pi nodes are required to meet ...
1
vote
0answers
853 views

Hadoop commands are taking a very long time to return

I am logged in (via SSH) to the NameNode of my Hadoop cluster; the problem I am having is that any hadoop fs commands, even simple ones like hadoop fs -ls are completed quickly, but take many minutes ...
1
vote
2answers
4k views

Hadoop DataNode is giving me an incompatible namespace ID

When I run the start-all.sh script from my master node, some of my DataNodes fail to start; the log file reports a Java IOException: Incompatible Namespace IDs in /tmp/$MY_USER_NAME.
1
vote
0answers
90 views

Increase CDH4 diskspace

Initially i was using bundled CDH3 VM (Cloudera Version of Hadoop) later, i removed CDH3 and now I am using CDH4 on CentOS as a VM (through VMWare player) with host machine as Win7 64bit. I am in the ...
0
votes
1answer
196 views

How much power supply do I need for my server, and could a shortage be causing my odd crashing?

I have 5 servers, all with similar hardware (i7, four 2tb 7200rpm drives, two 4tb 5400rpm drives, 430 watt power supply), and lately the machines have been freezing up. This has gotten worse in the ...
1
vote
1answer
817 views

Hadoop initscript askes password

I have installed hadoop on my ubuntu 12.04 single node .I am trying to execute an init script to make the hadoop run on start up but it asks password every time i execute. #!/bin/sh ### BEGIN ...
0
votes
3answers
7k views

Hadoop slave nodes not connecting

I have been trying to set up a Hadoop cluster; I managed to get it running in pseudo-distributed mode, and my one machine wordcounted Tolstoy's War and Peace in about thirty seconds. I am now trying ...
1
vote
1answer
598 views

Hadoop on Ubuntu - two different install directories?

I recently installed Hadoop 1.0.3 from the .deb provided by Apache. The package installed correctly, but there seem to be two directories that have Hadoop-related files: /usr/share/hadoop has jars and ...
3
votes
2answers
335 views

PXE boot Linux - which directories must be writable?

I'm planning to set up a small Hadoop cluster where the slave nodes boot and run from a central PXE server, to simplify deployment and updates, and to enable all of the disks on the slaves to be ...
2
votes
1answer
342 views

Setting up hive with hadoop

I am trying to set up the hive. I am using this guide: https://cwiki.apache.org/Hive/gettingstarted.html and I'm stuck at setting up /tmp and /user/hive/warehouse dirs. First of all, it seems to me a ...
0
votes
1answer
896 views

Empty homedir name when SSHing to localhost

I am trying to start a system (Hadoop, but that should not matter much for this question), and need to be able to ssh to localhost. I do this on windows with cygwin. The cygwin SSHD service is ...
0
votes
1answer
611 views

Access HDFS HADOOP using APACHE Web Server, Linux CentOS

If I have an apache web server as a directory how I can access to HDFS Cluster for upload and modify files, what is the configuration i want to do? many thanks
1
vote
1answer
531 views

What version of HDFS is compatible with HBase stable?

HBase stable is currently hbase-0.90.4, what version(s) of HDFS is it compatible with?
0
votes
2answers
551 views

I am trying to install Hadoop 0.23 but its not getting installed

I am trying to install Hadoop 0.23 on my fedora 13 but its not getting installed.On web every were there is support for hadoop 0.20 installation. I am trying installation from here. Every time i am ...
1
vote
1answer
207 views

scribe log analysis

I have decided to use scribe to log all the error and request details in my site for analysis. How can I use the scribe log data to analyze the data. Is there any tool for this or scribe server ...
1
vote
1answer
2k views

JAVA_HOME for Hadoop

I want to configure hadoop to run in pseudo-distributed mode. My configuration files: core-site.xml: <configuration> <property> <name>fs.default.name</name> ...
-2
votes
3answers
3k views

why http://server:60010 web page for the running HBase Master

I installed by : wget http://archive.cloudera.com/cdh/3/hbase-0.90.3-cdh3u1.tar.gz This is my hbase_site.xml <property> <name>hbase.master</name> ...
0
votes
1answer
235 views

What is the Ideal instance type for hadoop namenode

For a relatively small, one terabyte cluster ( 2TB actual after replication ) I was trying to nail down what the namenode's ideal memory/cpu size would be, having worked with hadoop off and on as an ...
3
votes
1answer
306 views

starting hadoop on mac os lion

I want to start hadoop on my macbook pro, I did all the steps that apache says. When I use the command "bin/start-all.sh", I get this: starting namenode, logging to ...
1
vote
0answers
33 views

Hadoop logging on Mac going to console.. any setting to have it go to Hadoop's log files instead?

When I run a command for Hadoop on my Mac, Hadoop sends the logs to the command line. I could capture this on my own with > mylogfile.log, but I'm hoping there is simply a setting/option in Hadoop ...
2
votes
1answer
97 views

Most secure way to issue commands on ubuntu cluster with sudo'ing user?

This is sort of a follow-up question to an unanswered question I have regarding administration of Cloudera cluster, but I figure generalizing the question to all of Ubuntu may help me get an answer. ...
0
votes
1answer
1k views

How do I grant a user permission to use Hadoop via Kerberos?

I've setup Hadoop to use Kerberos (following the Cloudera security guide), but it is unclear how I connect to hadoop with regular users (e.g. username=myuser). Currently I have myself ...
1
vote
1answer
805 views

Running hadoop jobs on Cloudera 3 as regular user?

Looking at Cloudera's installation instructions, I don't see any mention of how to run jobs as regular users. When I try to run a sample job, this is what I get: hadoop jar ...
1
vote
1answer
2k views

Are the Hadoop {start|stop}-all.sh scripts typically used w/ cloudera 3 clusters, and if so, how?

I've got a cluster I'm setting up Cloudera 3 on, and it is unclear on whether I should be using these start/stop scripts like I used to with the standard Apache Hadoop setup (where I had a specific ...
1
vote
0answers
199 views

why i am not able to install cdh3

https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3onUbuntuSystems I have ubuntu 8.04 ,i wget the squeeze and tried to dpkg it gives me this error . ...
5
votes
1answer
3k views

Is it possible to Managing 20 TB data using MySQL?

I am working in a project and my job is to build a database system to manage about 60,000,000,000 data entries. The project background is I have to do real-time storage for large number of messages ...
1
vote
0answers
813 views

Remote connection to Hadoop namenode issue

I am required to hack a single node hadoop "cluster" (cloudera psuedo-distributed) to be able to access it remotely. I have successfully installed hadoop and I have updated the localhost identifiers ...
3
votes
2answers
9k views

Hadoop HDFS: set file block size from commandline?

I need to set the block-size of a file when I load it into HDFS, to some value lower than the cluster block size. For example, if HDFS is using 64mb blocks, I may want a large file to be copied in ...
1
vote
1answer
2k views

Hive metadata permission issue

We are getting this error on Hive, while creating a DB / table hive> CREATE TABLE pokes (foo INT, bar STRING); FAILED: Error in metadata: ...