Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [hadoop]

Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

0
votes
0answers
14 views

Zookeeper best practice - using only SSD disks

we have HDP clusters version 2.6.0 / 2.6.1 ( hortonworks ) , and maybe in the future the version 3.0 I searched a lot in HORTONWORKS documentation , but not found the details about using SSD disks ...
0
votes
0answers
13 views

how to know the right swap size for hadoop Linux machines

we are using many hadoop clusters for now we are using swap size - 16G free -g total used free shared buff/cache available Mem: 125 ...
-1
votes
0answers
30 views

How can I import a MySQL table into Hive as an ORC file format using Sqoop?

I have a table into MySQL database and want to import it as a orc table in hive using sqoop. How can I write sqoop code for this ?
0
votes
0answers
9 views

How to configure Kerberos authentication on the browsers which are on CITRIX page?

We are connecting to our secure client network via CITRIX. We are using chrome to open all quick links. like ambari etc. They open and we are good there, but other useful links like RM and HISTORY ...
0
votes
0answers
32 views

Avoid kafka disk to became 100% used by Cron job

We want to suggest the following based on our issues on kafka disks We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 ) Each cluster include 3 kafka machines , ...
0
votes
1answer
21 views

Free Account Azure version HDInsight and cores issue

I am using an Free Azure account version and I am trying to create the resources needed to put in place HDInsight. I have done it twice, but in order to spare the time/money I have available, I have ...
1
vote
2answers
30 views

can we mix MTU values in cluster

we have hadoop cluster ( all machines are linux redhat machines version 7.x ) on the VM machines we set MTU=8900 and all other machines we set MTU=9000 we set on VM MTU=8900 because we saw some ...
0
votes
0answers
15 views

How to see if a Hadoop node has a job running on it

In the Jobtracker UI (resource manager), it is possible to see a list of all the jobs running on the cluster. However, it does not list the nodes where the job has running containers. Is there a way, ...
0
votes
0answers
71 views

How to debug Oozie action stuck in DONE state

I have an Oozie workflow that is used for all jobs and since few days some of the jobs submitted get stuck in RUNNING. Looking at the job the last action is in DONE state. oozie job -info 0014479-...
0
votes
2answers
51 views

IP Port same for source and destination

Yesterday, while debugging in a hadoop cluster i noticed something strange # netstat -taupen | grep 54310 tcp 0 0 10.0.12.209:54310 10.0.12.209:54310 TIME_WAIT You can notice that source ...
0
votes
0answers
217 views

metrics collector + Failed to invoke 'unbuffer' + socket connection left open in CLOSE_WAIT state

we have hadoop cluster on redhat machines , while metrics collector installed and running on one of the master machines we see strange warning from the logs , and we cant understand what the meaning ...
0
votes
1answer
45 views

Using same password for kerberos and openldap

We have a working structure for our hadop where openldap was used for authentication with below structure,along with ranger and knox. openldap root:- dn: dc=abchadoop,dc=com,dc=za Subtree inside ...
0
votes
0answers
151 views

HADOOP 3.1 multi nodes setup : where is the `hadoop-env.sh` file?

I want to install hadoop 3.1.0 on CentOS7.4i vms in multi node setup. So far I'm following strictly the official documentation, which is here. In the paragraph "Configuring the environment of hadoop ...
0
votes
0answers
8 views

Tez doesn't know the packaging structure of hadoop

My java project is packaged in a jar. The structure is like hadoop-0.0.1-SNAPSHOT-hadoop.jar |-- META-INF/ |-- com/ |-- lib/ `-- log4j.properties It works well ...
0
votes
0answers
89 views

How to add generated url in quicklinks of the apache ambari service?

I have added the a custom service of H2O in apache ambari as a hadoop job. Now that service is up and running. The thing is, when we start it, it starts the UI on any of the cluster ip which is not ...
0
votes
0answers
28 views

Searching for files within HDFS folder packed inside Tars/GZips?

Is there an easy way to search a directory in HDFS to list-only tar contents or gzip contents and then flag to console that it is inside this or that tar/gzip? Do I have to run some sort of python/...
1
vote
0answers
42 views

Sample output of Rumen or Input to Gridmix

I want to see JobHistory logs, which can be fed as input to the Rumen. More specifically, I am interested in knowing input format for the Gridmix. I tried following two things for it: 1) I found ...
0
votes
0answers
67 views

I am getting blow stack guard error while starting the name node service

I am getting blow stack guard error while starting the name node service on Pseudo Distributed installation mode. Starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-hadoop....
1
vote
1answer
80 views

Using posix attributes instead of normal LDAP?

Due to the way a software we use interacts with Unix, when I am setting up a certain application to interact with LDAP I need to use Posix attributes instead of normal LDAP attributes. So far all I ...
0
votes
1answer
32 views

Bare metal to Big Data: Can all of these operate together on the same cluster? [duplicate]

I am a VERY new sysadmin (Class of '16) and I've been asked to create a big data cluster with 3 bare metal PowerEdge Servers. I have the following request to be put on the cluster: *Hadoop2 *YARN *...
0
votes
0answers
243 views

Using OpenLDAP as back end for Kerberos

We want to integrate our existing security setup (Apache Knox, OpenLDAP, Apache Ranger) with Kerberos. So what I understood through some blogs that we can use OpenLDAP as a back-end for the Kerberos ...
0
votes
0answers
16 views

How to copy /dfs/nn from another node, without ssh access to the namenode

Is there any way to copy /dfs/nn directory on the namenode in HDFS to localsystem without accessing the namenode ? Can we do curl ?
0
votes
1answer
106 views

dose setting Jumbo frame on linux redhat influencing OS performance?

We set the MTU to 9000 on all our linux machines ( we have redhat machines version - 7.3 ) , linux machines are part of hadoop clusters we want to know if set the MTU to 9000 can be negative on OS ...
0
votes
1answer
245 views

how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster

we have ambari cluster ( version 2.6 ) with 3 workers machine , and each worker machine have 16 CPU CORE ( see pic down ) , while each machine have 32G memory according to: yarn.nodemanager.resource....
0
votes
0answers
29 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
3
votes
0answers
2k views

Spark Error: Failed to Send RPC to Datanode

We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now. From the log we can see an error message: Failed to send RPC 9053901149358924945 ...
0
votes
0answers
60 views

ambari + how to calculate the value of yarn.nodemanager.resource.cpu-vcores"

we have ambari cluster with the following machines: 3 masters machines 12 workers machines ( data-node machines ) , while each machine have 8 core the default value of yarn.nodemanager.resource....
0
votes
0answers
35 views

ambari cluster + when need to set Block replication to 1

We get the following in Spark logs: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
0
votes
0answers
95 views

is there any option to restore/recover the fsimage files ? on ambari cluster

we have a cluster ( ambari cluster with 3 master machines , 3 workers machines , 2 kafka's machines ) name-node is set on master01 and master03 Linux redhat machines version 7.3 we notice that after ...
0
votes
0answers
68 views

ambari cluster + No valid image files found

we have a old ambari cluster machine hadoop - version 2.6 all machines in the ambari cluster are redhat version - 7.3 we have a problem to start the standby name-node on the last master machine ...
0
votes
0answers
113 views

Hadoop cluster performance after Meltdown/Spectre kernel patch/upgrade

We run our Hadoop cluster on CentOS 7 and after patching server with new kernel patch 3.10.0-693.11.6.el7.x86_64 we are seeing 15% to 20% drop in CPU performance. We saw this when we ran terasort jobs....
0
votes
1answer
277 views

why ambari agent insist to create another repository file

we are installing the new hadoop version - 2.6.3.0 on ambari - 2.6.0 from ambari agent log we see the follwing: Writing File['/etc/yum.repos.d/ambari-hdp-51.repo'] because contents don't match why ...
0
votes
0answers
211 views

hadoop new version - HDP-2.6.3.0 failed because unnecessary repo file

we are trying to install the new hadoop version HDP version as scratch installation on our redhat machines ( redhat 7.3 version) HDP-2.6.3.0-centos7-rpm.tar.gz ambari-2.6.0.0-centos7.tar.gz before ...
1
vote
1answer
73 views

Redirecting Ambari-Server backup file creation to a different location

I am taking backup of my Ambari server using the command ambari-server backup This creates the backup file in the location /var/lib/ambari-server/ I want the backup to go to a different location, ...
0
votes
0answers
247 views

blueprint failed to installed the new version HDP - 2.6.3 on new version 2.6.x

we installed from scratch the HDP version - HDP - 2.6.3 & ambari 2.6.x our environment include 3 masters machines and 2 workers machines ( machines are redhat version 7.3 ) by using the ...
0
votes
0answers
34 views

Datanode machines disks size

is it important that ( workers ) datanode machines disks will be with the same size? for example we have ambari cluster with 3 workers machines ( datanode machines ) each datanode machine have 10 ...
1
vote
0answers
25 views

dose ambari cluster needs ssh access between ambari-server machine to all other host

We installed ambari cluster with 3 masters machines While ambari server installed on master02 linux machine ambari cluster also include 25 DataNodes machines and 5 kafka's machines dose ambari-...
1
vote
0answers
55 views

CDH Community Edition Upgrade from 5.7 to 5.13 without Cloudera Express or Cloudera Enterprise

I am having a cluster like below - 3 Hbase Master (1 Active & 2 standby) 4 Region Servers 4 Data Nodes 1 Primary & 1 Secondary Name Node 3 Journal Node 4 Nodemanager 3 Resource Manager (1 ...
0
votes
1answer
88 views

kafka + how to revert topic deletion

just in case we delete the wrong topic as /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper hdpmaster01:2181 --delete --topic gtom.poli.pri.proc Topic gtom.poli.pri.procis already ...
0
votes
0answers
623 views

how to force delete topic from kafka

we do the following steps in order to delete the topic - hgpo.llo.prmt.processed but even after 12 hours topics folders still not deleted from /var/kafka/kafka-logs note - we set - delete.topic....
0
votes
1answer
671 views

what is the safe and best way to delete the kafka topic folders

on all our kafka machines ( production machines ) , we see that: ( no free space ) df -h /var/kafka Filesystem Size Used Avail Use% Mounted on /dev/sdb 11T 11T 2.3M 100% /var/kafka ...
1
vote
1answer
498 views

Issues Programmatically Adding ODBC DSN to ODBC Administrator on Windows

I am working on trying to automate some configuration, and as part of that we need to add an ODBC DSN through a script. The driver I'm trying to use is the Cloudera Impala ODBC Connector, downloaded ...
0
votes
1answer
115 views

grafana + graphite + failed on test connection

first I'm new to Grafana so excuse me if i don't use the right words to describe my issue I installed Grafana + Graphite solution in my ambari cluster ( I am using ambari cluster with graphite ...
0
votes
1answer
256 views

what is effected when running - hadoop namenode -format

we have amabri cluster ( version 2.6 ) with 24 workers machines we want to run following commands only on worker23 machine ( because problem on worker23 ) , dose these commands effected on all ...
1
vote
0answers
38 views

How to prevent arbitrary executable execution on hadoop cluster

I am involved with configuring a Hadoop cluster for complete auditability and security. I am new to the Hadoop ecosystem, but I have a decent idea of the basics. I have a few concerns for which I hope ...
1
vote
2answers
1k views

How to identify and fix Missing blocks reported by Ambari for the NameNode?

Ambari is generating an alert NameNode Blocks Health: Total Blocks:[38252543], Missing Blocks:[2]. No further information. I've run hdfs fsck / which is reporting the entire filesystem as healthy. I'...
0
votes
1answer
729 views

calc md5 checksum for all files in hadoop directory

iam using hadoop apache 2.7.1 on centos and iam new to centos if i want to calc md5 checksum for specific file in hadoop i can issue the following command hdfs dfs -cat /hadoophome/myfile | ...
1
vote
0answers
108 views

YARN AM logs report different time-stamp from what is shown in terminal and sparkscala shell

I am trying to understand why the following occurred: I have a Docker container with Yarn and Spark running fine except that the timestamp of that container was minus X hours of what I wanted it to ...
-1
votes
1answer
1k views

Unable to ssh localhost without password despite proper perms, key in authorized_keys

I have a key ~/.ssh/id_rsa and I added the pub key to my authorized keys: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys I also changed my permissions to 600: sudo chmod 600 ~/.ssh/...
0
votes
1answer
164 views

How to reconfigure Ambari services values with blueprint.json file

we have many Ambari LAB clusters - Apache Ambari Version 2.5.0.3 , while ambari agent installed on Linux redhat machines my target is to find a way to update the values of services , on all the ...