Questions tagged [hadoop]
Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.
101
questions with no upvoted or accepted answers
6
votes
0
answers
10k
views
Spark Error: Failed to Send RPC to Datanode
We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now.
From the log we can see an error message:
Failed to send RPC 9053901149358924945 ...
3
votes
0
answers
167
views
How can I launch hdfs on Mesos without DC/OS?
From my understand DC/OS is a freemium managed service. Because I'd rather just have a raw Mesos implementation, I'd rather not be dependent on DC/OS and so I just want to know how to implement HDFS ...
2
votes
0
answers
392
views
Unable to start Apache Kylin
Good morning, I am trying to install Kylin-3.1.1 on a remote linux server, I made sure it had all software requirments and I installed already the following programms: apache-hive-3.1.2-bin , kylin-3....
2
votes
0
answers
3k
views
Yarn error: Failed to create Spark client for Spark session
I'm a bit new to this and have little experience, would appreciate your help.
I'm trying to install Hive on an existing Spark installation.
I mostly followed the instructions in this page with no ...
2
votes
2
answers
8k
views
How to identify and fix Missing blocks reported by Ambari for the NameNode?
Ambari is generating an alert NameNode Blocks Health: Total Blocks:[38252543], Missing Blocks:[2]. No further information.
I've run hdfs fsck / which is reporting the entire filesystem as healthy. I'...
2
votes
0
answers
4k
views
Hadoop Streaming with Python 3.5: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127
I'm trying to run my own mapper and reducer Python scripts using Hadoop Streaming on my cluster built on VMware Workstation VMs.
Hadoop version - 2.7, Python - 3.5, OS - CentOS 7.2 on all the VMs.
I ...
2
votes
0
answers
46
views
AWS-Hadoop Data Analytics Implementation for Multiple JSON Files
I am new to hadoop and AWS. I have setup multi-node (4 instances t2.large) AWS EC2 cluster with cloudera Hadoop distribution. I have tested the environment with basic examples using CSV files such as ...
2
votes
0
answers
522
views
Hbase block locality index is always 0
I have a Hbase (v 0.94.19 with Hadoop 1.2.1) setup with one master machine and two region servers. Each region server has 16 GB heap (6.4 GB cache, 4.0 GB memstore) and 1.6 TB (2 X 800 GB) SSD disk ...
2
votes
0
answers
2k
views
Encounter an error when configuer secure hadoop : org.apache.hadoop.security.AccessControlException
I try to configure secure hadoop with kerberos. I have started KDC server , generated and copy related keytab to corresponding node. kerberos can work normally (use kinit)
but when i try to start ...
2
votes
1
answer
896
views
Hadoop - What is the purpose of the /usr/sbin/ shell scripts?
I am installing Hadoop 1.1.2 on CentOS 6.4.
I read all the Hadoop documentation at http://hadoop.apache.org/docs/stable/
After installing, I noticed there are many shell scripts at /usr/sbin/. But ...
2
votes
2
answers
1k
views
Hadoop initscript askes password
I have installed hadoop on my ubuntu 12.04 single node .I am trying to execute an init script to make the hadoop run on start up but it asks password every time i execute.
#!/bin/sh
### BEGIN ...
1
vote
0
answers
170
views
HDP cluster + journal nodes get out of Sync
we have HDP cluster version 2.6.5
when we look on name-node logs we can see the following warning
2023-02-20 15:56:37,731 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(...
1
vote
0
answers
35
views
YARN + how to debug wget
we are testing with wget VIA port 8088 the connection from ResourceManager02 to ResourceManager01
both Resource Managers are part of YARN service , and each resource manager service installed on RHEL ...
1
vote
0
answers
11
views
How to debug policy enforcement on YARN queues?
I have a HDP 3.1 cluster and it seems that the fair policy isn't behaving as expected or YARN is misconfigured, since some users/applications/jobs are consuming more resources than we supposed it to ...
1
vote
0
answers
72
views
Backup and Restore strategy in Hbase cluster
I have just started with Hbase cluster. I have a Hbase cluster with 2 master nodes and 4 slave nodes. I have one hbase table where huge data is populated everyday so the disk gets filled quickly. I ...
1
vote
0
answers
39
views
How to install hadoop reusable on a local network?
For a multi-node image cluster on an institute, we have sevaral laptops and machines and we want to create a hadoop cluster with hbase on the top for indexing the data/images.
I have tried some VMare ...
1
vote
1
answer
289
views
Hadoop: Failed to start backup node, bad state: DROP_UNTIL_NEXT_ROLL
I have created a small Hadoop cluster setup with 1 NameNode and 1 DataNode to get hands-on.
below is my configuration files:
Core-site.xml
<property>
<name>fs.defaultFS</name>
...
1
vote
1
answer
383
views
Zookeeper best practice - using only SSD disks
we have HDP clusters version 2.6.0 / 2.6.1 ( hortonworks ) , and maybe in the future the version 3.0
I searched a lot in HORTONWORKS documentation , but not found the details about using SSD disks ...
1
vote
0
answers
101
views
Sample output of Rumen or Input to Gridmix
I want to see JobHistory logs, which can be fed as input to the Rumen. More specifically, I am interested in knowing input format for the Gridmix.
I tried following two things for it:
1) I found ...
1
vote
1
answer
2k
views
Using posix attributes instead of normal LDAP?
Due to the way a software we use interacts with Unix, when I am setting up a certain application to interact with LDAP I need to use Posix attributes instead of normal LDAP attributes.
So far all I ...
1
vote
0
answers
63
views
spark.dynamicAllocation + setting the spark parameters according to ambari cluster
we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,,
spark....
1
vote
0
answers
191
views
ambari cluster + when need to set Block replication to 1
We get the following in Spark logs:
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
1
vote
0
answers
31
views
dose ambari cluster needs ssh access between ambari-server machine to all other host
We installed ambari cluster with 3 masters machines
While ambari server installed on master02 linux machine
ambari cluster also include 25 DataNodes machines and 5 kafka's machines
dose ambari-...
1
vote
0
answers
95
views
CDH Community Edition Upgrade from 5.7 to 5.13 without Cloudera Express or Cloudera Enterprise
I am having a cluster like below -
3 Hbase Master (1 Active & 2 standby)
4 Region Servers
4 Data Nodes
1 Primary & 1 Secondary Name Node
3 Journal Node
4 Nodemanager
3 Resource Manager (1 ...
1
vote
1
answer
497
views
kafka + how to revert topic deletion
just in case we delete the wrong topic as
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper hdpmaster01:2181 --delete --topic gtom.poli.pri.proc
Topic gtom.poli.pri.procis already ...
1
vote
0
answers
56
views
How to prevent arbitrary executable execution on hadoop cluster
I am involved with configuring a Hadoop cluster for complete auditability and security. I am new to the Hadoop ecosystem, but I have a decent idea of the basics. I have a few concerns for which I hope ...
1
vote
0
answers
403
views
YARN AM logs report different time-stamp from what is shown in terminal and sparkscala shell
I am trying to understand why the following occurred:
I have a Docker container with Yarn and Spark running fine except that the timestamp of that container was minus X hours of what I wanted it to ...
1
vote
0
answers
20
views
Need to set 000 permission to specific hdfs data block through commandline
I am trying to set the “000” permission to the specific block.
I used below command to find the block information:
su - hdfs -c "hdfs fsck -locations -files -blocks /user/rohit/...
1
vote
0
answers
142
views
Why Hadoop TeraSort not using all cluster nodes
Question
Regarding the TeraSort demo in hadoop, please suggest if the symptom is as expected or the workload should be distributed.
Symptom
Started Hadoop (3 nodes in a cluster) and run the ...
1
vote
0
answers
27
views
Is it possible to configure hdfs in a federation mode and in an HA mode in the same time?
I don't understand if it possible to configure HDFS in both modes in the same time. Does it make sense? Can somebody show a simple configuration of HDFS in both modes? (nameNode1, nameNode2, ...
1
vote
0
answers
875
views
Zombie process blocking port when restarting Hadoop (Secondary) Namenode
I'm having weird issues with the Hadoop Namenode and Secondary Namenode.
Our HDFS cluster runs smoothly most of the time. But every now and then, either the Primary Namenode freezes (crashing the ...
1
vote
0
answers
32
views
Hadoop Hive, Impala, Pig, and more — SQL access to Hadoop?
It appears that Hive, Impala, Pig, and others all provide SQL or SQL-like access to data stored on Hadoop clusters. They all seem to have support for HDFS, S3, and other forms.
So why are there so ...
1
vote
1
answer
1k
views
hadoop fs commands not giving any output
I am running hadoop 1.2.1 on ubuntu 14.04 LTS in pseudo distributed mode, and the fs commands are not doing anything, i.e. niether prompt is returned nor any error message. what is the problem? Thanks ...
1
vote
1
answer
295
views
Hyper-V VLAN with static IP
Trying to setup/simulate Hadoop cluster locally via multiple (hyper-v) VMs.
I'm using hortonworks sandbox image for hyper-v which runs centos linux.
I can hit the vm, if I use an internal switch and ...
1
vote
1
answer
1k
views
Hadoop namenode port getting blocked
I have installed 7 VM instances of Ubuntu 14.04 LTS servers. First instance runs the namenode service and all other 6 nodes run datanode service.I think my NameNode is getting crashed or blocked due ...
1
vote
0
answers
562
views
How to force HDFS to use LDAP user's UID
I have a cloudera cluster with HDFS and Hue services and I'm trying to unify the authentication using LDAP.
I have my LDAP server running thanks to 389-ds (not sure if is the best way) and I can log ...
1
vote
0
answers
75
views
When and how are initial directories created in HDFS
I have a Hadoop setup in which the configured HDFS umask is 027 instead of the default one. Some of the initially created directories have correct permissions (like tmp drwxrwxrwx) but others such as /...
1
vote
1
answer
566
views
Additional Storage Options for Hadoop HDFS Nodes
We have a small production Cloudera distribution Hadoop cluster(14 nodes, but growing). As we have expanded our usage of this cluster we have found that disk storage is our biggest blocker and ...
1
vote
1
answer
823
views
Changing ulimit on ubuntu 12.04 Never works
I am working with hadoop and need to change number of open files ulimit -n. I have seen similar questions on stackoverflow and elsewhere and have tried everything in those answers, but it still does ...
1
vote
0
answers
2k
views
Use Amazon SNS to send nagios alerts
Is there any way by which I can send nagios alerts to Amazon SNS ?
I have tried with following steps, but its giving me this error in the Nagios log file:
Jul 12 11:38:23 ip-10-134-13-204 nagios3: ...
1
vote
1
answer
322
views
Java process failure (hadoop, hbase)
Anytime I am running hadoop/hbase process from a command prompt I get an error:
/usr/local/hadoop/bin/hadoop: line 320: /usr/lib/jvm/jdk1.7.0/bin/java: cannot execute binary file
/usr/local/hadoop/...
1
vote
1
answer
331
views
hadoop: port appears open locally but not remotelly
I am new to linux and hadoop and I am having the same issue as in this question. I think I understand what is causing it but I don't know how to solve it (Don't know what they mean by "Edit the ...
1
vote
1
answer
243
views
Data lost after Hdfs client was killed
I wrote a simple tool to upload logs to HDFS. And I found some curious phenomenon.
If I run the tool in foreground and close it with "Ctrl - C", there will be some data in HDFS.
If I run the tool in ...
1
vote
0
answers
2k
views
Hadoop commands are taking a very long time to return
I am logged in (via SSH) to the NameNode of my Hadoop cluster; the problem I am having is that any hadoop fs commands, even simple ones like hadoop fs -ls are completed quickly, but take many minutes ...
1
vote
0
answers
101
views
Increase CDH4 diskspace
Initially i was using bundled CDH3 VM (Cloudera Version of Hadoop) later, i removed CDH3 and now I am using CDH4 on CentOS as a VM (through VMWare player) with host machine as Win7 64bit. I am in the ...
1
vote
0
answers
62
views
Hadoop logging on Mac going to console.. any setting to have it go to Hadoop's log files instead?
When I run a command for Hadoop on my Mac, Hadoop sends the logs to the command line. I could capture this on my own with > mylogfile.log, but I'm hoping there is simply a setting/option in Hadoop ...
1
vote
0
answers
242
views
why i am not able to install cdh3
https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3onUbuntuSystems
I have ubuntu 8.04 ,i wget the squeeze and tried to dpkg it gives me this error .
ksn@ksn-...
1
vote
0
answers
1k
views
Remote connection to Hadoop namenode issue
I am required to hack a single node hadoop "cluster" (cloudera psuedo-distributed) to be able to access it remotely. I have successfully installed hadoop and I have updated the localhost identifiers ...
1
vote
0
answers
93
views
Hadoop machines going down - logs to look for?
I have a hadoop cluster with ~7 machines, and some of the machines were keep going down. Sometimes, the hadoop datanode / jobtracker processes only dies (the machine is still running), and other times,...
1
vote
2
answers
2k
views
NoNode for HBase master pseudodistributed mode
I am using Ubuntu 18.04, hadoop 3.1.3 and hbase 2.2.1
To me it seems like my hadoop and HBase are not configured correctly to interact. When I through the HBase shell try to create a table it yields ...