The tag has no wiki summary.

learn more… | top users | synonyms

6
votes
2answers
66 views

HBASE Space Used Started Climbing Rapidly

Update 4,215: After looking at space usage inside of hdfs, I see that .oldlogs is using a lot of space: 1485820612766 /hbase/.oldlogs So new questions: What is it? How do I clean it up? How ...
-1
votes
0answers
66 views

HDFS setup with Cloudera: DataNode fails to connect to NameNode

I have 3 DataNodes and 1 NameNode on a machine inside LXC containers. The DataNode on the same node as the NameNode works fine but the other 2 don't i get: Initialization failed for Block pool ...
1
vote
0answers
77 views

How to force HDFS to use LDAP user's UID

I have a cloudera cluster with HDFS and Hue services and I'm trying to unify the authentication using LDAP. I have my LDAP server running thanks to 389-ds (not sure if is the best way) and I can log ...
2
votes
1answer
46 views

Linux Network tuning to prevent tcp rcvpruned and backlogdrop?

My datanodes in my hbase cluster are triggering some tcp rcvpruned and backlog drops from time to time: It seems to be there are at least two angles to approach this at: Tune HBase/HDFS etc... so ...
1
vote
1answer
111 views

Rhadoop hdfs.init() Error

I recently installed CDH5.1.0 along with R 3.1.*, and I got rmr2, rJava, and rhdfs all installed properly. (along with the required packages and set the required environment variables) After some ...
1
vote
0answers
536 views

“The volume for a file has been externally altered …” and setting file sizes on write to NFS from Windows

I have set up NFS access to HDFS. I can connect to the default root share from both Windows and Linux clients and all clients can read from the share. Writing from the Linux client is also ok. When ...
2
votes
2answers
336 views

Disable The Under Replicated Blocks Alert in Cloudera Manager

I have a single server Hbase cluster that I am only using as the sink end of HBase replication. Therefore I don't want to replicate any blocks within this cluster (since the source has replicated ...
0
votes
2answers
559 views

error while running any hadoop hdfs file system command

I am very new to hadoop and referring to the "hadoop for dummies" book. I have a VM with following specs: hadoop version 2.0.6-alpha bigtop os centos The problem is when I run any hdfs file system ...
1
vote
1answer
270 views

Unable to convert HDFS from non-HA to HA

Introduction Aim: Convert HDFS from non-HA to HA. Method: According to this documentation it should be possible to convert HFDS from non- to HA by implementing following configuration: ...
2
votes
0answers
536 views

java.lang.NullPointerException When Doing A Read in HDFS

I have had a 10 node HBase cluster up and running for the past 4 months. The cluster was setup on VMs in a corporate environment which I do not control, but everything has been working great...until ...
2
votes
2answers
216 views

In Hadoop, how to show current process of -copyFromLocal

I am still a newbie learner of Hadoop, and this time I was trying to process a 106GB file. I used -copyFromLocal to copy that big file to my Hadoop DFS, but since the file is big I have to wait for a ...
1
vote
0answers
20 views

When and how are initial directories created in HDFS

I have a Hadoop setup in which the configured HDFS umask is 027 instead of the default one. Some of the initially created directories have correct permissions (like tmp drwxrwxrwx) but others such as ...
0
votes
1answer
470 views

btrfs: Looking for experience on btrfs error modes on bad sectors / read errors / medium erros

While running a Hadoop cluster that uses HDFS (so data is already replicated 3x) we experience often issues with ext4 because some bad sectors (the filesystem is unable to read or write to them) ...
1
vote
1answer
166 views

Additional Storage Options for Hadoop HDFS Nodes

We have a small production Cloudera distribution Hadoop cluster(14 nodes, but growing). As we have expanded our usage of this cluster we have found that disk storage is our biggest blocker and ...
0
votes
1answer
279 views

Change existing linux user's uid in LDAP but hdfs seems does not recognize it

I have set up a hadoop 1.2.1 environment on centos 6. I also use nfs-proxy that mount ndfs to local file system so that I can access the files inside hdfs locally. It works perfectly until today, I ...
0
votes
2answers
590 views

HDFS datanode startup fails when disks are full

Our HDFS cluster is only 90% full but some datanodes have some disks that are 100% full. That means when we mass reboot the entire cluster some datanodes completely fail to start with a message like ...
2
votes
2answers
1k views

Hadoop HDFS Backup & DR Strategy

We are preparing to implement our first Hadoop cluster. As such we are starting out small with a four node setup. (1 master node, and 3 worker nodes) Each node will have 6TB of storage. (6 x 1TB ...
3
votes
0answers
3k views

How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?

I have a 5 slave Hadoop cluster (using CDH4)---slaves are where DataNode and TaskNode run. Each slave has 4 partitions dedicated to HDFS storage. One of the slaves needed a reinstall and this caused ...
2
votes
1answer
1k views

Setting up permissions in Cloudera Hadoop 4 (CDH4)

I would like to set up HDFS permissions in CDH4, with the following requirements: Everyone can read everything from all HDFS directories Each user can only write to his user directory on HDFS Except ...
1
vote
1answer
70 views

Data lost after Hdfs client was killed

I wrote a simple tool to upload logs to HDFS. And I found some curious phenomenon. If I run the tool in foreground and close it with "Ctrl - C", there will be some data in HDFS. If I run the tool in ...
1
vote
1answer
722 views

Hadoop hdfs namenode is throwing an error

Full list of error: hb@localhost:/etc/hadoop/conf$ sudo service hadoop-hdfs-namenode start * Starting Hadoop namenode: starting namenode, logging to ...
3
votes
1answer
6k views

Can't connect to HDFS in pseudo-distributed mode

I followed the instructions here for installing hadoop in pseudo-distributed mode. However, I'm having trouble connecting to HDFS. When I execute this command : ./hadoop fs -ls / I get a ...
2
votes
1answer
2k views

Deleting temp directory from HDFS

Is there a smart way of deleting old files from the hdfs /tmp directory? (Just to make sure, I am not talking about the unix FS /tmp)
1
vote
0answers
290 views

Hadoop commands are taking a very long time to return

I am logged in (via SSH) to the NameNode of my Hadoop cluster; the problem I am having is that any hadoop fs commands, even simple ones like hadoop fs -ls are completed quickly, but take many minutes ...
1
vote
2answers
2k views

Hadoop DataNode is giving me an incompatible namespace ID

When I run the start-all.sh script from my master node, some of my DataNodes fail to start; the log file reports a Java IOException: Incompatible Namespace IDs in /tmp/$MY_USER_NAME.
1
vote
1answer
342 views

What version of HDFS is compatible with HBase stable?

HBase stable is currently hbase-0.90.4, what version(s) of HDFS is it compatible with?
1
vote
1answer
404 views

Processing pre-existing log files with Flume

I have a large set of log files that I need to extract data from. Is it possible to use Flume to read these files and dump them into an HDFS (Cassandra, or another data source) which I can then ...
3
votes
2answers
5k views

Hadoop HDFS: set file block size from commandline?

I need to set the block-size of a file when I load it into HDFS, to some value lower than the cluster block size. For example, if HDFS is using 64mb blocks, I may want a large file to be copied in ...
2
votes
1answer
604 views

Does changing the default HDFS replication factor from 3 affect mapper performance?

Have a HDFS/Hadoop cluster setup and am looking into tuning. I wonder if changing the default HDFS replication factor (default:3) to something bigger will improve mapper performance, at the obvious ...
4
votes
1answer
617 views

Ceph: Why is a greater number of “placement groups” a “bad thing”?

I have been researching distributed databases and file systems, and while I was originally mostly interested in Hadoop/HBase because I'm a Java programmer, I found this very interesting document about ...
0
votes
1answer
1k views

Compiling hdfs-fuse bundled with Hadoop

I am trying to compile the hdfs-fuse extension from Hadoop 0.20.2 on a machine running Fedora 14. Below are the packages I have installed: fuse-2.8.5-2.fc14.x86_64 fuse-libs-2.8.5-2.fc14.x86_64 ...
0
votes
1answer
180 views

hdfs configuration

I am a newbie. Trying to setup a hdfs system to serve my data (I don't plan to use mapreduce) at my lab. So far I have read, cluster setup in but I am still confused. Several questions: Do I need ...
5
votes
1answer
560 views

Forward-sync to HDFS? (OR continue an incomplete hdfs upload?)

Anyone have a good suggestion for doing a forward sync to HDFS? ("forward-sync" in contrast to "bi-directional sync") Basically I have a large number of files I want to put into the HDFS. Its so ...
1
vote
3answers
4k views

What is meant by “streaming data access” in HDFS?

According to the HDFS Architecture page HDFS was designed for "streaming data access". I'm not sure what that means exactly, but would guess it means an operation like seek is either disabled or has ...