Questions tagged [hadoop]

Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Enable read-only HDFS filesystem in hostile cluster

Scenario A company's platform team maintains Spark clusters for data science teams to perform big data analysis on their data. The cluster comes with a pre-configured file system that allows read/...
2
votes
1answer
43 views

kafka confluent vs Apache kafka

for now we have HDP 2.6.5 version , with Kafka cluster with Kafka version 0.1 all Kafka machines are installed on RHEL version 7.6 all machines with RHEL 7.6 are on DELL physical machines we want to ...
0
votes
0answers
10 views

Hadoop Environment Simplification

It had been seven years since I last installed a multi-cluster Apache Hadoop node, so it was naturally a bit nostalgic for me to implement one today. In the times since I started using linux-based ...
0
votes
0answers
45 views

How to mount conda/miniconda to a NFS Mount in GCP Dataproc

I have a requirement to mount /opt/conda to an NFS mount, So all the worker nodes would have the same environments created. This would also be helpful in autoscaling worker nodes, as all the new ...
0
votes
1answer
16 views

Hadoop recommissioning datanode

Do I need to delete all data from a datanode before recommissioning it, or it doesn't matter and the namenode will not pick stale data from the datanode?
2
votes
0answers
52 views

Unable to start Apache Kylin

Good morning, I am trying to install Kylin-3.1.1 on a remote linux server, I made sure it had all software requirments and I installed already the following programms: apache-hive-3.1.2-bin , kylin-3....
0
votes
0answers
115 views

apache livy session state “starting” and app id “null”

Apache Livy installed on HDP cluster's node which have HDP clients installed. Also the same node have spark configured too. Spark version 2.3.3 Livy version 0.6 and 0.7 (tried both the versions) hdp....
1
vote
0answers
24 views

Backup and Restore strategy in Hbase cluster

I have just started with Hbase cluster. I have a Hbase cluster with 2 master nodes and 4 slave nodes. I have one hbase table where huge data is populated everyday so the disk gets filled quickly. I ...
1
vote
1answer
68 views

what's the meaning of Requested resource=<memory:-1, vCores:1> in hadoop web ui?

I want to insert some data in to hive table. but it is stuck. So I go to the hadoop web ui and find the following information: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: ...
0
votes
1answer
27 views

Change HDFS replication factor

I've changed replication factor from 3 to 2 for some directories with command: hdfs dfs -setrep -R 2 /path/to/dir but my HDFS free space still the same. Should I do something else to free my disks?
0
votes
1answer
27 views

HDFS. How to free 1 particular disk

I have cluster with 3 servers. 2 of them have 2 TB disks and another one have 500 Gb SSD. I am trying to use balancer, but I still get 70% of usage on 2TB disks and 99% on 500Gb due to non-dfs files. ...
0
votes
0answers
19 views

hadoop cluster + what are the configuration steps in order to change the domain name on machines cluster

We have HDP cluster version 2.6.5 and ambari 2.6.1 version Cluster include 3 masters machines , and 211 data-nodes machines ( workers machines ) , all machines are rhel 7.2 version Example master1....
0
votes
0answers
29 views

How to decide Zookeeper's snapshot file policy?

ZooKeeper server creates snapshot and log files, but never deletes them. So we need to care about the retention policy. How to decide on the right amount of remaining ZooKeeper snapshot files? Need to ...
0
votes
1answer
76 views

Hadoop Cluster Capacity Planning of Data Nodes for disks per data node

we are planing to build hadoop cluster with 12 data nodes machines when the replication factor is 3 and DataNode failed disk tolerance - 1 data nodes machines are include the disks for HDFS since we ...
0
votes
0answers
19 views

cannot access hdfs even though hadoop is (apparently) installed on CentOS server

I'm attempting to install SparkBeagle https://github.com/NGSeq/SparkBeagle on a CentOS server, following instructions on https://linuxconfig.org/how-to-install-hadoop-on-redhat-8-linux. I've opened an ...
0
votes
1answer
375 views

What is the default password of hive

With local console, typing "hive" launching the console directly without any password. But, when I try to connect using dbeaver/beeline, it prompts for username/password. I tried with hive/&...
0
votes
1answer
225 views

Optimal RAID configuration for EC2 instance store used for HDFS

I'm trying to determine if there is any practical advantage to configuring a RAID array on the instance store of a 3x d2.2xlarge instances being used for HDFS. Initially I planned to just mount each ...
1
vote
1answer
1k views

NoNode for HBase master pseudodistributed mode

I am using Ubuntu 18.04, hadoop 3.1.3 and hbase 2.2.1 To me it seems like my hadoop and HBase are not configured correctly to interact. When I through the HBase shell try to create a table it yields ...
0
votes
1answer
15 views

Users for Hadoop deployment

I followed instructions found online to install and configuring a "2 name nodes" + "10 data nodes" hadoop cluster on CentOS 8. I created a wheel user called "hadoop" on all nodes and setup ...
1
vote
1answer
500 views

List all files in hdfs directory

Due to some error at one component, files in HDFS got accumulated and the number is huge i.e 2123516. I want to list all files and want to copy their name in one file but when I run the following ...
0
votes
1answer
205 views

mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk

We need to create xfs file-system on kafka disk The special thing about kafka disk is the disk size kafka disk have 20TB size in our case I not sure about the following mkfs , but I need advice to ...
0
votes
1answer
130 views

is it possible mix different RHEL OS version in hadoop cluster?

we are using the following HDP cluster with ambari , list of nodes and their RHEL version 3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2 312 DATA-NODES machines ...
0
votes
0answers
92 views

Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?

I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
0
votes
0answers
32 views

Request Time Out / Sessions Stalling through IPTABLE (DNAT)

Scenario: Customer recently Migrated Clustered HANA DB Servers to Azure Cloud Platform but these are Physical Servers on Azure (Offering: Azure HLI). Usually these HLIs (HANA DB Servers) in Azure ...
0
votes
1answer
26 views

transferring data between two hadoop clusters without direct network connectivity

I have a need to transfer data fairly regularly (on demand, not scripted / streamed) between two independent hadoop clusters. One of which is deployed in an isolated network and has no direct access ...
2
votes
0answers
2k views

Yarn error: Failed to create Spark client for Spark session

I'm a bit new to this and have little experience, would appreciate your help. I'm trying to install Hive on an existing Spark installation. I mostly followed the instructions in this page with no ...
0
votes
1answer
80 views

hadoop + can we install zookeeper servers on kafka hosts

we want to dedicated the zookeeper servers only for kafka machines so each kafka machine include the zookeeper server and zookeeper server will serve only the kafka host and not other application in ...
0
votes
2answers
974 views

High Active(file) Memory Usage in Oracle Linux VMs

I recently searched and read lots of posts and questions about Linux memory management but I can't find my case. For example, there is a question in Unix StackExchange about High memory usage but no ...
1
vote
0answers
31 views

How to install hadoop reusable on a local network?

For a multi-node image cluster on an institute, we have sevaral laptops and machines and we want to create a hadoop cluster with hbase on the top for indexing the data/images. I have tried some VMare ...
0
votes
1answer
343 views

HDFS balancing , how to balanced hdfs data?

we have Hadoop version - 2.6.4 On the datanode machine we can see that hdfs data isn’t balanced On some disks we have different used size as sdb 11G and sdd 17G /dev/sdd 20G 3.0G 17G 15% /grid/...
1
vote
0answers
143 views

Hadoop: Failed to start backup node, bad state: DROP_UNTIL_NEXT_ROLL

I have created a small Hadoop cluster setup with 1 NameNode and 1 DataNode to get hands-on. below is my configuration files: Core-site.xml <property> <name>fs.defaultFS</name> ...
0
votes
1answer
186 views

Install Nvidia Drivers 9.0 for TensorFlow pip (Debian 9.7)

I installed Nvidia drivers 9.1 on my Debian 9.7 (Dataproc) when I try to run TensorFlow 1.9 via this test script it fails: Used this guide to install GPU Drivers: https://cloud.google.com/dataproc/...
1
vote
1answer
263 views

Zookeeper best practice - using only SSD disks

we have HDP clusters version 2.6.0 / 2.6.1 ( hortonworks ) , and maybe in the future the version 3.0 I searched a lot in HORTONWORKS documentation , but not found the details about using SSD disks ...
0
votes
1answer
76 views

How to configure Kerberos authentication on the browsers which are on CITRIX page?

We are connecting to our secure client network via CITRIX. We are using chrome to open all quick links. like ambari etc. They open and we are good there, but other useful links like RM and HISTORY ...
0
votes
0answers
48 views

Avoid kafka disk to became 100% used by Cron job

We want to suggest the following based on our issues on kafka disks We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 ) Each cluster include 3 kafka machines , ...
0
votes
1answer
29 views

Free Account Azure version HDInsight and cores issue

I am using an Free Azure account version and I am trying to create the resources needed to put in place HDInsight. I have done it twice, but in order to spare the time/money I have available, I have ...
1
vote
2answers
69 views

can we mix MTU values in cluster

we have hadoop cluster ( all machines are linux redhat machines version 7.x ) on the VM machines we set MTU=8900 and all other machines we set MTU=9000 we set on VM MTU=8900 because we saw some ...
0
votes
2answers
1k views

IP Port same for source and destination

Yesterday, while debugging in a hadoop cluster i noticed something strange # netstat -taupen | grep 54310 tcp 0 0 10.0.12.209:54310 10.0.12.209:54310 TIME_WAIT You can notice that source ...
0
votes
1answer
250 views

Using same password for kerberos and openldap

We have a working structure for our hadop where openldap was used for authentication with below structure,along with ranger and knox. openldap root:- dn: dc=abchadoop,dc=com,dc=za Subtree inside ...
1
vote
0answers
96 views

Sample output of Rumen or Input to Gridmix

I want to see JobHistory logs, which can be fed as input to the Rumen. More specifically, I am interested in knowing input format for the Gridmix. I tried following two things for it: 1) I found ...
1
vote
1answer
1k views

Using posix attributes instead of normal LDAP?

Due to the way a software we use interacts with Unix, when I am setting up a certain application to interact with LDAP I need to use Posix attributes instead of normal LDAP attributes. So far all I ...
0
votes
1answer
34 views

Bare metal to Big Data: Can all of these operate together on the same cluster? [duplicate]

I am a VERY new sysadmin (Class of '16) and I've been asked to create a big data cluster with 3 bare metal PowerEdge Servers. I have the following request to be put on the cluster: *Hadoop2 *YARN *...
0
votes
0answers
492 views

Using OpenLDAP as back end for Kerberos

We want to integrate our existing security setup (Apache Knox, OpenLDAP, Apache Ranger) with Kerberos. So what I understood through some blogs that we can use OpenLDAP as a back-end for the Kerberos ...
0
votes
1answer
273 views

dose setting Jumbo frame on linux redhat influencing OS performance?

We set the MTU to 9000 on all our linux machines ( we have redhat machines version - 7.3 ) , linux machines are part of hadoop clusters we want to know if set the MTU to 9000 can be negative on OS ...
0
votes
1answer
2k views

how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster

we have ambari cluster ( version 2.6 ) with 3 workers machine , and each worker machine have 16 CPU CORE ( see pic down ) , while each machine have 32G memory according to: yarn.nodemanager.resource....
1
vote
0answers
53 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
5
votes
0answers
7k views

Spark Error: Failed to Send RPC to Datanode

We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now. From the log we can see an error message: Failed to send RPC 9053901149358924945 ...
1
vote
0answers
157 views

ambari cluster + when need to set Block replication to 1

We get the following in Spark logs: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
0
votes
1answer
673 views

why ambari agent insist to create another repository file

we are installing the new hadoop version - 2.6.3.0 on ambari - 2.6.0 from ambari agent log we see the follwing: Writing File['/etc/yum.repos.d/ambari-hdp-51.repo'] because contents don't match why ...
1
vote
1answer
88 views

Redirecting Ambari-Server backup file creation to a different location

I am taking backup of my Ambari server using the command ambari-server backup This creates the backup file in the location /var/lib/ambari-server/ I want the backup to go to a different location, ...

1
2 3 4 5 6