Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

0
votes
1answer
1k views

Can't start Hadoop from an init.d script

I'm using CentOS 6.2. I'm trying to start Hadoop from an init.d script, but it's failing. This is what I see in boot.log : Retrigger failed udev events [ OK ] ...
2
votes
2answers
2k views

Deleting temp directory from HDFS

Is there a smart way of deleting old files from the hdfs /tmp directory? (Just to make sure, I am not talking about the unix FS /tmp)
0
votes
1answer
111 views

Hadoop Configuration Files - Who Needs What?

As I am setting up Hadoop, one question keeps popping in my mind but I can't find the answer. Which Hadoop configuration files need to be copied to which nodes. For example, I'm making changes to the ...
0
votes
1answer
142 views

Hadoop CDH4 Evaluation: Which Ubuntu would be preferred? Lucid or Precise [closed]

I'm setting up a CDH4 in AWS for evaluation (we already have a CHD3 running on Ubuntu Lucid) and I'd like advice regarding any known gotchyas that I'd be likely to encounter if running it on Lucid vs. ...
3
votes
1answer
588 views

Deploy Hadoop to Openstack

I'd like to deploy Hadoop to Openstack cloud. Is there any automatic way to do that? Anyone tried to do that? I'm looking for some devops like juju. I've never used juju and right now I'm going ...
1
vote
1answer
3k views

Raspberry Pi based Hadoop cluster [closed]

Is it at least possible to build Hadoop cluster from Raspberry Pi-based nodes? Can such a cluster meet hardware requirements of Hadoop? And if so, how much Raspberry Pi nodes are required to meet ...
1
vote
0answers
520 views

Hadoop commands are taking a very long time to return

I am logged in (via SSH) to the NameNode of my Hadoop cluster; the problem I am having is that any hadoop fs commands, even simple ones like hadoop fs -ls are completed quickly, but take many minutes ...
1
vote
2answers
3k views

Hadoop DataNode is giving me an incompatible namespace ID

When I run the start-all.sh script from my master node, some of my DataNodes fail to start; the log file reports a Java IOException: Incompatible Namespace IDs in /tmp/$MY_USER_NAME.
1
vote
0answers
84 views

Increase CDH4 diskspace

Initially i was using bundled CDH3 VM (Cloudera Version of Hadoop) later, i removed CDH3 and now I am using CDH4 on CentOS as a VM (through VMWare player) with host machine as Win7 64bit. I am in the ...
0
votes
1answer
179 views

How much power supply do I need for my server, and could a shortage be causing my odd crashing?

I have 5 servers, all with similar hardware (i7, four 2tb 7200rpm drives, two 4tb 5400rpm drives, 430 watt power supply), and lately the machines have been freezing up. This has gotten worse in the ...
1
vote
1answer
592 views

Hadoop initscript askes password

I have installed hadoop on my ubuntu 12.04 single node .I am trying to execute an init script to make the hadoop run on start up but it asks password every time i execute. #!/bin/sh ### BEGIN ...
0
votes
3answers
5k views

Hadoop slave nodes not connecting

I have been trying to set up a Hadoop cluster; I managed to get it running in pseudo-distributed mode, and my one machine wordcounted Tolstoy's War and Peace in about thirty seconds. I am now trying ...
1
vote
1answer
591 views

Hadoop on Ubuntu - two different install directories?

I recently installed Hadoop 1.0.3 from the .deb provided by Apache. The package installed correctly, but there seem to be two directories that have Hadoop-related files: /usr/share/hadoop has jars and ...
3
votes
2answers
309 views

PXE boot Linux - which directories must be writable?

I'm planning to set up a small Hadoop cluster where the slave nodes boot and run from a central PXE server, to simplify deployment and updates, and to enable all of the disks on the slaves to be ...
2
votes
1answer
314 views

Setting up hive with hadoop

I am trying to set up the hive. I am using this guide: https://cwiki.apache.org/Hive/gettingstarted.html and I'm stuck at setting up /tmp and /user/hive/warehouse dirs. First of all, it seems to me a ...
0
votes
1answer
799 views

Empty homedir name when SSHing to localhost

I am trying to start a system (Hadoop, but that should not matter much for this question), and need to be able to ssh to localhost. I do this on windows with cygwin. The cygwin SSHD service is ...
0
votes
1answer
464 views

Access HDFS HADOOP using APACHE Web Server, Linux CentOS

If I have an apache web server as a directory how I can access to HDFS Cluster for upload and modify files, what is the configuration i want to do? many thanks
1
vote
1answer
427 views

What version of HDFS is compatible with HBase stable?

HBase stable is currently hbase-0.90.4, what version(s) of HDFS is it compatible with?
0
votes
2answers
543 views

I am trying to install Hadoop 0.23 but its not getting installed

I am trying to install Hadoop 0.23 on my fedora 13 but its not getting installed.On web every were there is support for hadoop 0.20 installation. I am trying installation from here. Every time i am ...
1
vote
1answer
199 views

scribe log analysis

I have decided to use scribe to log all the error and request details in my site for analysis. How can I use the scribe log data to analyze the data. Is there any tool for this or scribe server ...
1
vote
1answer
1k views

JAVA_HOME for Hadoop

I want to configure hadoop to run in pseudo-distributed mode. My configuration files: core-site.xml: <configuration> <property> <name>fs.default.name</name> ...
-2
votes
3answers
2k views

why http://server:60010 web page for the running HBase Master

I installed by : wget http://archive.cloudera.com/cdh/3/hbase-0.90.3-cdh3u1.tar.gz This is my hbase_site.xml <property> <name>hbase.master</name> ...
0
votes
1answer
220 views

What is the Ideal instance type for hadoop namenode

For a relatively small, one terabyte cluster ( 2TB actual after replication ) I was trying to nail down what the namenode's ideal memory/cpu size would be, having worked with hadoop off and on as an ...
3
votes
1answer
288 views

starting hadoop on mac os lion

I want to start hadoop on my macbook pro, I did all the steps that apache says. When I use the command "bin/start-all.sh", I get this: starting namenode, logging to ...
1
vote
0answers
31 views

Hadoop logging on Mac going to console.. any setting to have it go to Hadoop's log files instead?

When I run a command for Hadoop on my Mac, Hadoop sends the logs to the command line. I could capture this on my own with > mylogfile.log, but I'm hoping there is simply a setting/option in Hadoop ...
2
votes
1answer
92 views

Most secure way to issue commands on ubuntu cluster with sudo'ing user?

This is sort of a follow-up question to an unanswered question I have regarding administration of Cloudera cluster, but I figure generalizing the question to all of Ubuntu may help me get an answer. ...
0
votes
1answer
1k views

How do I grant a user permission to use Hadoop via Kerberos?

I've setup Hadoop to use Kerberos (following the Cloudera security guide), but it is unclear how I connect to hadoop with regular users (e.g. username=myuser). Currently I have myself ...
1
vote
1answer
778 views

Running hadoop jobs on Cloudera 3 as regular user?

Looking at Cloudera's installation instructions, I don't see any mention of how to run jobs as regular users. When I try to run a sample job, this is what I get: hadoop jar ...
1
vote
1answer
2k views

Are the Hadoop {start|stop}-all.sh scripts typically used w/ cloudera 3 clusters, and if so, how?

I've got a cluster I'm setting up Cloudera 3 on, and it is unclear on whether I should be using these start/stop scripts like I used to with the standard Apache Hadoop setup (where I had a specific ...
1
vote
0answers
189 views

why i am not able to install cdh3

https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3onUbuntuSystems I have ubuntu 8.04 ,i wget the squeeze and tried to dpkg it gives me this error . ...
4
votes
1answer
3k views

Is it possible to Managing 20 TB data using MySQL?

I am working in a project and my job is to build a database system to manage about 60,000,000,000 data entries. The project background is I have to do real-time storage for large number of messages ...
1
vote
0answers
722 views

Remote connection to Hadoop namenode issue

I am required to hack a single node hadoop "cluster" (cloudera psuedo-distributed) to be able to access it remotely. I have successfully installed hadoop and I have updated the localhost identifiers ...
3
votes
2answers
7k views

Hadoop HDFS: set file block size from commandline?

I need to set the block-size of a file when I load it into HDFS, to some value lower than the cluster block size. For example, if HDFS is using 64mb blocks, I may want a large file to be copied in ...
1
vote
1answer
2k views

Hive metadata permission issue

We are getting this error on Hive, while creating a DB / table hive> CREATE TABLE pokes (foo INT, bar STRING); FAILED: Error in metadata: ...
0
votes
1answer
266 views

Hadoop + NAT scenario

I have a situation where I'd like to run Hadoop spread across 2 clusters. The first cluster (ClusterA) is normal and all nodes are publicly accessible. The second cluster (ClusterB) is behind a NAT. ...
1
vote
0answers
55 views

Hadoop machines going down - logs to look for?

I have a hadoop cluster with ~7 machines, and some of the machines were keep going down. Sometimes, the hadoop datanode / jobtracker processes only dies (the machine is still running), and other ...
4
votes
1answer
3k views

Hadoop: Blacklisted tasktracker

I am running a Hadoop job (using Hadoop 0.20.2) on a 6 machine setup; one machine is the namenode / secondary node / job tracker (master) and the other 5 machines are all datanodes / tasktrackers ...
2
votes
1answer
838 views

Does changing the default HDFS replication factor from 3 affect mapper performance?

Have a HDFS/Hadoop cluster setup and am looking into tuning. I wonder if changing the default HDFS replication factor (default:3) to something bigger will improve mapper performance, at the obvious ...
0
votes
1answer
489 views

Hadoop streaming job on EC2 stays in “pending” state

Trying to experiment with Hadoop and Streaming using cloudera distribution CDH3 on Ubuntu. Have valid data in hdfs:// ready for processing. Wrote little streaming mapper in python. When I launch a ...
1
vote
1answer
192 views

problem while running apache mahout quickstartryin

I was trying to run the mahout clustering example from the quickstart at: https://cwiki.apache.org/MAHOUT/clustering-of-synthetic-control-data.html While running any of the clustering ...
0
votes
1answer
518 views

Datanode not showing in WEB interface

Newbie on hadoop clusters. I have setup my two nodes conf as described by M. G. Noll here. The datanode has datanode & tasktracker running (jps command show them). However in the WEB UI I only ...
2
votes
3answers
104 views

Hadoop moving data to another user

I have a few hundreds of GB in my hdfs for userA (single node configuration). I would like to transfer all that data to userB wich will be more appropriate for the multi-node configuration I'm setting ...
0
votes
2answers
264 views

Adding smaller nodes to pseudo-distributed nutch/hadoop cluster

I have nutch/hadoop pseudo distributed running fine. I want to add processing capacity by adding new nodes which are smaller than master (HD 3 times smaller) and cheaper of course. Since the default ...
2
votes
4answers
7k views

Hadoop JBOD disk configuration on HP Smart Array 410/i disk controller

I'm in a evaluation phase of some hw that could be used for setting up a hadoop cluster. This hw is refurbished (hp G6 servers w/ Smart Array 410/i controller) and probably we should/must use it... we ...
0
votes
1answer
132 views

Need help to build a strategy

I am a Junior System Administrator with one of the Engineering Schools. One of the Professors got a donation of 45 servers (Dell Poweredge 1690) from Yahoo. Following are his requirements: hadoop ...
1
vote
2answers
535 views

Does hadoop take care of different node HD size alone?

I have a single node (pseudo-distributed config) and I'm considering adding a 2nd slave node. Does it matter if the slave has less HD capacity ? Will the rebalance take of that for itself. I'm not an ...
0
votes
2answers
283 views

How to use combined CPU/Memory power of a Windows cluster

I have 5 Windows machines (dual-core, 3GB) in a LAN all joined to a domain. I have a program which needs 8-cores and 10 GB to run in a given SLA time. What platform/tool can i use to harness the ...
2
votes
1answer
816 views

Is there a way to get a list of Hadoop cluster machines from one of the data nodes?

I have access to a data node in a Hadoop cluster, and I'd like to find out the identity of the name nodes for the same cluster. Is there a way to do this?
3
votes
1answer
639 views

Best practice for administering a (hadoop) cluster

I've recently been playing with Hadoop. I have a six node cluster up and running - with HDFS, and having run a number of MapRed jobs. So far, so good. However I'm now looking to do this more ...
0
votes
1answer
2k views

Compiling hdfs-fuse bundled with Hadoop

I am trying to compile the hdfs-fuse extension from Hadoop 0.20.2 on a machine running Fedora 14. Below are the packages I have installed: fuse-2.8.5-2.fc14.x86_64 fuse-libs-2.8.5-2.fc14.x86_64 ...