Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

learn more… | top users | synonyms

0
votes
1answer
1k views

How do I grant a user permission to use Hadoop via Kerberos?

I've setup Hadoop to use Kerberos (following the Cloudera security guide), but it is unclear how I connect to hadoop with regular users (e.g. username=myuser). Currently I have myself ...
1
vote
1answer
717 views

Running hadoop jobs on Cloudera 3 as regular user?

Looking at Cloudera's installation instructions, I don't see any mention of how to run jobs as regular users. When I try to run a sample job, this is what I get: hadoop jar ...
1
vote
1answer
1k views

Are the Hadoop {start|stop}-all.sh scripts typically used w/ cloudera 3 clusters, and if so, how?

I've got a cluster I'm setting up Cloudera 3 on, and it is unclear on whether I should be using these start/stop scripts like I used to with the standard Apache Hadoop setup (where I had a specific ...
1
vote
0answers
167 views

why i am not able to install cdh3

https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3onUbuntuSystems I have ubuntu 8.04 ,i wget the squeeze and tried to dpkg it gives me this error . ...
4
votes
1answer
2k views

Is it possible to Managing 20 TB data using MySQL?

I am working in a project and my job is to build a database system to manage about 60,000,000,000 data entries. The project background is I have to do real-time storage for large number of messages ...
1
vote
0answers
596 views

Remote connection to Hadoop namenode issue

I am required to hack a single node hadoop "cluster" (cloudera psuedo-distributed) to be able to access it remotely. I have successfully installed hadoop and I have updated the localhost identifiers ...
2
votes
2answers
4k views

Hadoop HDFS: set file block size from commandline?

I need to set the block-size of a file when I load it into HDFS, to some value lower than the cluster block size. For example, if HDFS is using 64mb blocks, I may want a large file to be copied in ...
0
votes
1answer
1k views

Hive metadata permission issue

We are getting this error on Hive, while creating a DB / table hive> CREATE TABLE pokes (foo INT, bar STRING); FAILED: Error in metadata: ...
0
votes
1answer
183 views

Hadoop + NAT scenario

I have a situation where I'd like to run Hadoop spread across 2 clusters. The first cluster (ClusterA) is normal and all nodes are publicly accessible. The second cluster (ClusterB) is behind a NAT. ...
1
vote
0answers
52 views

Hadoop machines going down - logs to look for?

I have a hadoop cluster with ~7 machines, and some of the machines were keep going down. Sometimes, the hadoop datanode / jobtracker processes only dies (the machine is still running), and other ...
3
votes
1answer
3k views

Hadoop: Blacklisted tasktracker

I am running a Hadoop job (using Hadoop 0.20.2) on a 6 machine setup; one machine is the namenode / secondary node / job tracker (master) and the other 5 machines are all datanodes / tasktrackers ...
2
votes
1answer
518 views

Does changing the default HDFS replication factor from 3 affect mapper performance?

Have a HDFS/Hadoop cluster setup and am looking into tuning. I wonder if changing the default HDFS replication factor (default:3) to something bigger will improve mapper performance, at the obvious ...
0
votes
1answer
332 views

Hadoop streaming job on EC2 stays in “pending” state

Trying to experiment with Hadoop and Streaming using cloudera distribution CDH3 on Ubuntu. Have valid data in hdfs:// ready for processing. Wrote little streaming mapper in python. When I launch a ...
1
vote
1answer
172 views

problem while running apache mahout quickstartryin

I was trying to run the mahout clustering example from the quickstart at: https://cwiki.apache.org/MAHOUT/clustering-of-synthetic-control-data.html While running any of the clustering ...
0
votes
1answer
312 views

Datanode not showing in WEB interface

Newbie on hadoop clusters. I have setup my two nodes conf as described by M. G. Noll here. The datanode has datanode & tasktracker running (jps command show them). However in the WEB UI I only ...
2
votes
3answers
100 views

Hadoop moving data to another user

I have a few hundreds of GB in my hdfs for userA (single node configuration). I would like to transfer all that data to userB wich will be more appropriate for the multi-node configuration I'm setting ...
0
votes
2answers
234 views

Adding smaller nodes to pseudo-distributed nutch/hadoop cluster

I have nutch/hadoop pseudo distributed running fine. I want to add processing capacity by adding new nodes which are smaller than master (HD 3 times smaller) and cheaper of course. Since the default ...
2
votes
4answers
5k views

Hadoop JBOD disk configuration on HP Smart Array 410/i disk controller

I'm in a evaluation phase of some hw that could be used for setting up a hadoop cluster. This hw is refurbished (hp G6 servers w/ Smart Array 410/i controller) and probably we should/must use it... we ...
0
votes
1answer
126 views

Need help to build a strategy

I am a Junior System Administrator with one of the Engineering Schools. One of the Professors got a donation of 45 servers (Dell Poweredge 1690) from Yahoo. Following are his requirements: hadoop ...
1
vote
2answers
349 views

Does hadoop take care of different node HD size alone?

I have a single node (pseudo-distributed config) and I'm considering adding a 2nd slave node. Does it matter if the slave has less HD capacity ? Will the rebalance take of that for itself. I'm not an ...
0
votes
2answers
240 views

How to use combined CPU/Memory power of a Windows cluster

I have 5 Windows machines (dual-core, 3GB) in a LAN all joined to a domain. I have a program which needs 8-cores and 10 GB to run in a given SLA time. What platform/tool can i use to harness the ...
2
votes
1answer
446 views

Is there a way to get a list of Hadoop cluster machines from one of the data nodes?

I have access to a data node in a Hadoop cluster, and I'd like to find out the identity of the name nodes for the same cluster. Is there a way to do this?
3
votes
1answer
578 views

Best practice for administering a (hadoop) cluster

I've recently been playing with Hadoop. I have a six node cluster up and running - with HDFS, and having run a number of MapRed jobs. So far, so good. However I'm now looking to do this more ...
0
votes
1answer
1k views

Compiling hdfs-fuse bundled with Hadoop

I am trying to compile the hdfs-fuse extension from Hadoop 0.20.2 on a machine running Fedora 14. Below are the packages I have installed: fuse-2.8.5-2.fc14.x86_64 fuse-libs-2.8.5-2.fc14.x86_64 ...
-1
votes
2answers
334 views

Hadoop on Virtual Machines

We would like to migrate from MySQL to Hadoop for scalability. Will hadoop clusters running on virtual machines improve our website's performance? What are the advantages and disadvantages of ...
0
votes
1answer
419 views

Can overriding of -Xmx be prevented for hadoop jobs?

I have a shared cluster running Hadoop-0.20.2. Occasionally users don't realize that the default memory settings chosen are based on the amount of available memory. Can I enforce a maximum value for ...
2
votes
1answer
318 views

Management of available file descriptors within a Hadoop cluster

I'm currently in charge of a rapidly-growing Hadoop cluster for my employer, currently built upon release 0.21.0 with CentOS as the OS for each worker and master node. I've worked through most of the ...
4
votes
1answer
898 views

Can a hadoop job be paused or suspended?

I'm using hadoop-0.20.2. Looking at hadoop fs. I am able to kill or fail an individual task. Is there anyway to pause it so that the map slots are freed up for another task?
1
vote
3answers
744 views

Hadoop Rolling Small files

I am running Hadoop on a project and need a suggestion. Generally by default Hadoop has a "block size" of around 64mb.. There is also a suggestion to not use many/small files.. I am currently having ...
1
vote
1answer
716 views

hadoop installation on VMware work station with multiple virtual machines

Hi This is what I want to do : I want to learn about hadoop and run some hands on distributed computing by some programming I have a windows box with window 7 professional installed in it, I also have ...
3
votes
4answers
478 views

Interpreting exim log files after parsing

I'm parsing exim log files and, due to my processing method, lose the original order of all entries in this file. I rebuild the transactions by their transaction ID (i.e. 1OfiYX-0000Ev-7k) but still ...
0
votes
4answers
899 views

Disks for hadoop, what do you recommend? [closed]

what is you recommendation about disks for Hadoop? Do you recommend using SAS, or just attaching disk over SATA? Or maybe something else? What are pros and cons of every option? (Decision about disk ...
2
votes
2answers
186 views

Pairing MySQL and NoSQL Solutions

We have some fairly large datasets (user events and server log information - >100 GB) that's becoming fairly unwieldy for data processing. I've seen lots of activity around NoSQL/Hadoop/etc and I was ...
1
vote
2answers
283 views

Even data distribution on hadoop/hive

I am trying a small hadoop setup (for experimentation) with just 2 machines. I am loading about 13GB of data, a table of around 39 million rows, with a replication factor of 1 using Hive. My problem ...
6
votes
2answers
2k views

Hadoop disk fail, what do you do?

I would like to know about your strategies on what to do when one of the Hadoop server disk fails. Let's say, I have multiple (>15) Hadoop servers and 1 namenode, and one from 6 disks on slaves ...
2
votes
1answer
2k views

Is there any way to use arrays in a puppet module (not in template)?

I want to use puppet to manage a hadoop cluster. On the machines we have several directories which must be created and set permissions. But i'm unable to add array values for defined methods. define ...
2
votes
1answer
569 views

Hadoop slaves file necessary?

I'm working on a team trying to create a system for creating Hadoop clusters on EC2 with minimal effort on the part of the user. Ideally, we would like slave instances to only require the hostname of ...
0
votes
1answer
7k views

/usr/bin/env: python2.5: No such file or directory

I'm trying to play with Cloudera's Distribution for Hadoop on my EC2 account. For configuring it i'm using THIS tutorial. Everything seems to me fine but when I'm trying to run hadoop-ec2 I'm ...
0
votes
4answers
266 views

Big Data: Which HD Parameters are Important?

I work with a lot of datasets that are in the tens of GBs, usually split into several files. Performing any type of dataside-wide operation (grep, sed, search, read/write to/from databases and Hadoop) ...
5
votes
1answer
500 views

Forward-sync to HDFS? (OR continue an incomplete hdfs upload?)

Anyone have a good suggestion for doing a forward sync to HDFS? ("forward-sync" in contrast to "bi-directional sync") Basically I have a large number of files I want to put into the HDFS. Its so ...
0
votes
1answer
428 views

gitosis interferes with hadoop

Never thought I'd write a title like that, but it's true I have gitosis and hadoop installed. > sudo /usr/lib/hadoop/bin/start-all.sh Enter passphrase for key '/root/.ssh/id_rsa': ...
1
vote
2answers
2k views

Any good method for mounting Hadoop HDFS from another system?

I want to mount the Cloudera Hadoop as a Linux file system over the LAN. As a setup, I already have the hadoop cluster running on a set of Ubuntu machines. But now I need to be able to use it as a ...
1
vote
2answers
405 views

hadoop/hive metastore

where do people place their multi user meta store? I'm going to use mysql but I don't know were I should stick it. on the name node or on its own server?
1
vote
3answers
3k views

What is meant by “streaming data access” in HDFS?

According to the HDFS Architecture page HDFS was designed for "streaming data access". I'm not sure what that means exactly, but would guess it means an operation like seek is either disabled or has ...
2
votes
2answers
2k views

hadoop - datanode decommission

I want to remove nodes from my cluster gracefully. I added the following to my hadoop-site.xml: <property> <name>dfs.hosts.exclude</name> ...
22
votes
3answers
15k views

What is Hadoop and what is it used for? [closed]

I have been enjoying reading ServerFault for a while and I have come across quite a few topics on Hadoop. I have had a little trouble finding out what it does from a global point of view. So my ...
4
votes
4answers
1k views

Hadoop cluster. 2 Fast, 4 Medium, 8 slower machines?

We're going to purchase some new hardware to use just for a Hadoop cluster and we're stuck on what we should purchase. Say we have a budget of $5k should we buy two super nice machines at $2500/each, ...