Questions tagged [hadoop]

Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

Filter by
Sorted by
Tagged with
0
votes
0answers
24 views

Push mongodb collections to Amazon redshift

I have got a mongodb database. I need to stream or push data from mongodb to Amazon redshift as and when data arrives in mongodb. Is there any solution for this..? Can this be implemented using Apache ...
1
vote
1answer
257 views

NoNode for HBase master pseudodistributed mode

I am using Ubuntu 18.04, hadoop 3.1.3 and hbase 2.2.1 To me it seems like my hadoop and HBase are not configured correctly to interact. When I through the HBase shell try to create a table it yields ...
0
votes
0answers
28 views

HDFS on docker containers - is it possible?

we are using Hadoop cluster based on HDP version - 2.6.5 with ambari platform we want to know if we can use the following HDFS components on docker containers as: journal nodes ZK fail controller ...
0
votes
0answers
32 views

Trying to figure out long run times with sqooping jobs (sql server to hadoop)

I inherited a new customer as a sql server dba and they are using some java-based framework that has a jdbc connection from sql server to hadoop. They have a sqooping job that runs once a day to do ...
0
votes
0answers
7 views

Users for Hadoop deployment

I followed instructions found online to install and configuring a "2 name nodes" + "10 data nodes" hadoop cluster on CentOS 8. I created a wheel user called "hadoop" on all nodes and setup ...
0
votes
0answers
125 views

List all files in hdfs directory

Due to some error at one component, files in HDFS got accumulated and the number is huge i.e 2123516. I want to list all files and want to copy their name in one file but when I run the following ...
0
votes
0answers
48 views

HDFS compatibility with different of disks size on datanode machines

We have HDP cluster version 2.6.5 , with data node machines ( HDFS ) hdfs version Hadoop 2.7.3.2.6.5.0-292 Subversion git@github.com:hortonworks/hadoop.git -r ...
0
votes
1answer
85 views

mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk

We need to create xfs file-system on kafka disk The special thing about kafka disk is the disk size kafka disk have 20TB size in our case I not sure about the following mkfs , but I need advice to ...
0
votes
0answers
93 views

Spark SQL throws an connection exception error?

I have installed Apache Hadoop,spark,hive(meta store as embedded derby) and Scala on a Ubuntu server and configured it individually.Whenever i try to run spark-sql it throws up a error stating 19/...
0
votes
0answers
17 views

Cloudera node /etc/krb5.conf replaced at every reboot

Why are my cloudera nodes replacing the file /etc/krb5.conf at every reboot? I'm trying to make modifications, and when someone issues a reboot the file is again replaced by the old config file. ...
0
votes
1answer
62 views

is it possible mix different RHEL OS version in hadoop cluster?

we are using the following HDP cluster with ambari , list of nodes and their RHEL version 3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2 312 DATA-NODES machines ...
0
votes
0answers
17 views

how to recover HDP cluster by installing HDP from scratch and save data-node disks and kafka disks data

I want to ask this important question lets say we have the following: HDP cluster with : 3 masters machine ( active/standby name-node ) , ( active/standby resource manager ) 3 datanode machines ...
0
votes
0answers
35 views

Can't access my Ambari Dashboard after the system shutdown abnormally

I have a master node and other 3 nodes, everything was working fine. There was a power interruption and all the 4 nodes went down and on restarting I can't reach my dashboard using the IP Address. ...
0
votes
0answers
54 views

Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?

I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
0
votes
0answers
18 views

Run shell script one per node on Hadoop cluster

I'm trying to automate some administrative tasks on the Hadoop cluster. Some of these tasks require that I run some shell commands on each node in the cluster (e.g. move the file to the predefined ...
0
votes
0answers
8 views

How to allow user to access other users policies in Hortonworks ranger

How to allow other users to see my resource based policies in Horotonworks ranger. I have included other users in permissions -> resource based policy When user click on any of the resource based ...
0
votes
0answers
16 views

DLP: How to restrict user from downloading data through Putty, SSH, SCP, Sqoop etc from Hadoop cluster

We need to enforce some controls to restrict users the capability of downloading data from hadoop via a variety of channels below (least but not all): Putty/SSH SCP Sqoop 1/2 Spark 1/2 Hue/Impala Hue/...
0
votes
0answers
30 views

Request Time Out / Sessions Stalling through IPTABLE (DNAT)

Scenario: Customer recently Migrated Clustered HANA DB Servers to Azure Cloud Platform but these are Physical Servers on Azure (Offering: Azure HLI). Usually these HLIs (HANA DB Servers) in Azure ...
0
votes
1answer
25 views

transferring data between two hadoop clusters without direct network connectivity

I have a need to transfer data fairly regularly (on demand, not scripted / streamed) between two independent hadoop clusters. One of which is deployed in an isolated network and has no direct access ...
2
votes
0answers
2k views

Yarn error: Failed to create Spark client for Spark session

I'm a bit new to this and have little experience, would appreciate your help. I'm trying to install Hive on an existing Spark installation. I mostly followed the instructions in this page with no ...
0
votes
1answer
52 views

hadoop + can we install zookeeper servers on kafka hosts

we want to dedicated the zookeeper servers only for kafka machines so each kafka machine include the zookeeper server and zookeeper server will serve only the kafka host and not other application in ...
0
votes
0answers
237 views

No such file or directory when I execute ( hdfs namenode -format )

After setting everything on the CentOS server to run Hadoop. When I execute hdfs namenode -format I get this result: hdfs namenode -format /home/wael/hadoop/bin/hdfs: line 304: /usr/lib/jvm/java-1.8....
0
votes
2answers
348 views

High Active(file) Memory Usage in Oracle Linux VMs

I recently searched and read lots of posts and questions about Linux memory management but I can't find my case. For example, there is a question in Unix StackExchange about High memory usage but no ...
1
vote
0answers
31 views

How to install hadoop reusable on a local network?

For a multi-node image cluster on an institute, we have sevaral laptops and machines and we want to create a hadoop cluster with hbase on the top for indexing the data/images. I have tried some VMare ...
0
votes
1answer
179 views

HDFS balancing , how to balanced hdfs data?

we have Hadoop version - 2.6.4 On the datanode machine we can see that hdfs data isn’t balanced On some disks we have different used size as sdb 11G and sdd 17G /dev/sdd 20G 3.0G 17G 15% /grid/...
1
vote
0answers
97 views

Hadoop: Failed to start backup node, bad state: DROP_UNTIL_NEXT_ROLL

I have created a small Hadoop cluster setup with 1 NameNode and 1 DataNode to get hands-on. below is my configuration files: Core-site.xml <property> <name>fs.defaultFS</name> ...
0
votes
1answer
179 views

Install Nvidia Drivers 9.0 for TensorFlow pip (Debian 9.7)

I installed Nvidia drivers 9.1 on my Debian 9.7 (Dataproc) when I try to run TensorFlow 1.9 via this test script it fails: Used this guide to install GPU Drivers: https://cloud.google.com/dataproc/...
1
vote
1answer
180 views

Zookeeper best practice - using only SSD disks

we have HDP clusters version 2.6.0 / 2.6.1 ( hortonworks ) , and maybe in the future the version 3.0 I searched a lot in HORTONWORKS documentation , but not found the details about using SSD disks ...
0
votes
1answer
59 views

How to configure Kerberos authentication on the browsers which are on CITRIX page?

We are connecting to our secure client network via CITRIX. We are using chrome to open all quick links. like ambari etc. They open and we are good there, but other useful links like RM and HISTORY ...
0
votes
0answers
46 views

Avoid kafka disk to became 100% used by Cron job

We want to suggest the following based on our issues on kafka disks We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 ) Each cluster include 3 kafka machines , ...
0
votes
1answer
25 views

Free Account Azure version HDInsight and cores issue

I am using an Free Azure account version and I am trying to create the resources needed to put in place HDInsight. I have done it twice, but in order to spare the time/money I have available, I have ...
1
vote
2answers
60 views

can we mix MTU values in cluster

we have hadoop cluster ( all machines are linux redhat machines version 7.x ) on the VM machines we set MTU=8900 and all other machines we set MTU=9000 we set on VM MTU=8900 because we saw some ...
0
votes
2answers
631 views

IP Port same for source and destination

Yesterday, while debugging in a hadoop cluster i noticed something strange # netstat -taupen | grep 54310 tcp 0 0 10.0.12.209:54310 10.0.12.209:54310 TIME_WAIT You can notice that source ...
0
votes
1answer
170 views

Using same password for kerberos and openldap

We have a working structure for our hadop where openldap was used for authentication with below structure,along with ranger and knox. openldap root:- dn: dc=abchadoop,dc=com,dc=za Subtree inside ...
1
vote
0answers
88 views

Sample output of Rumen or Input to Gridmix

I want to see JobHistory logs, which can be fed as input to the Rumen. More specifically, I am interested in knowing input format for the Gridmix. I tried following two things for it: 1) I found ...
1
vote
1answer
590 views

Using posix attributes instead of normal LDAP?

Due to the way a software we use interacts with Unix, when I am setting up a certain application to interact with LDAP I need to use Posix attributes instead of normal LDAP attributes. So far all I ...
0
votes
1answer
34 views

Bare metal to Big Data: Can all of these operate together on the same cluster? [duplicate]

I am a VERY new sysadmin (Class of '16) and I've been asked to create a big data cluster with 3 bare metal PowerEdge Servers. I have the following request to be put on the cluster: *Hadoop2 *YARN *...
0
votes
0answers
419 views

Using OpenLDAP as back end for Kerberos

We want to integrate our existing security setup (Apache Knox, OpenLDAP, Apache Ranger) with Kerberos. So what I understood through some blogs that we can use OpenLDAP as a back-end for the Kerberos ...
0
votes
1answer
250 views

dose setting Jumbo frame on linux redhat influencing OS performance?

We set the MTU to 9000 on all our linux machines ( we have redhat machines version - 7.3 ) , linux machines are part of hadoop clusters we want to know if set the MTU to 9000 can be negative on OS ...
0
votes
1answer
1k views

how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster

we have ambari cluster ( version 2.6 ) with 3 workers machine , and each worker machine have 16 CPU CORE ( see pic down ) , while each machine have 32G memory according to: yarn.nodemanager.resource....
1
vote
0answers
45 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
5
votes
0answers
6k views

Spark Error: Failed to Send RPC to Datanode

We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now. From the log we can see an error message: Failed to send RPC 9053901149358924945 ...
1
vote
0answers
105 views

ambari cluster + when need to set Block replication to 1

We get the following in Spark logs: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
0
votes
1answer
564 views

why ambari agent insist to create another repository file

we are installing the new hadoop version - 2.6.3.0 on ambari - 2.6.0 from ambari agent log we see the follwing: Writing File['/etc/yum.repos.d/ambari-hdp-51.repo'] because contents don't match why ...
1
vote
1answer
85 views

Redirecting Ambari-Server backup file creation to a different location

I am taking backup of my Ambari server using the command ambari-server backup This creates the backup file in the location /var/lib/ambari-server/ I want the backup to go to a different location, ...
0
votes
0answers
36 views

Datanode machines disks size

is it important that ( workers ) datanode machines disks will be with the same size? for example we have ambari cluster with 3 workers machines ( datanode machines ) each datanode machine have 10 ...
1
vote
0answers
28 views

dose ambari cluster needs ssh access between ambari-server machine to all other host

We installed ambari cluster with 3 masters machines While ambari server installed on master02 linux machine ambari cluster also include 25 DataNodes machines and 5 kafka's machines dose ambari-...
1
vote
0answers
82 views

CDH Community Edition Upgrade from 5.7 to 5.13 without Cloudera Express or Cloudera Enterprise

I am having a cluster like below - 3 Hbase Master (1 Active & 2 standby) 4 Region Servers 4 Data Nodes 1 Primary & 1 Secondary Name Node 3 Journal Node 4 Nodemanager 3 Resource Manager (1 ...
1
vote
1answer
258 views

kafka + how to revert topic deletion

just in case we delete the wrong topic as /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper hdpmaster01:2181 --delete --topic gtom.poli.pri.proc Topic gtom.poli.pri.procis already ...
0
votes
1answer
845 views

what is the safe and best way to delete the kafka topic folders

on all our kafka machines ( production machines ) , we see that: ( no free space ) df -h /var/kafka Filesystem Size Used Avail Use% Mounted on /dev/sdb 11T 11T 2.3M 100% /var/kafka ...

1
2 3 4 5 6