Questions tagged [hadoop]

Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

Filter by
Sorted by
Tagged with
0 votes
0 answers
55 views

YARN resource manager "HTTP request sent, awaiting response..."

we have Hadoop cluster with active/stand by resource manager services the active resource manager is on master1 machine and the stand by resource manager is on master2 machine in our cluster YARN ...
King David's user avatar
0 votes
0 answers
31 views

PostgreSQL + Removing old Host from DB not succeeded

We have HDP cluster with 152 workers machines - worker1.duplex.com .. worker152.duplex.com , While all machines are installed on RHEL 7.9 version We are trying to delete the last host - worker152....
King David's user avatar
0 votes
0 answers
30 views

Hadoop Namenode heap size tuning

NameNode process is executed in Java virtual machine, and Java object which NameNode creates is managed in Java virtual memory. As the fles or directories are created, inode objects and block objects ...
King David's user avatar
0 votes
0 answers
20 views

EOF Exception whnen runnign spark-shell on Hadoop Cluster

The problem I am trying to install Spark on Hadoop cluster. I've installed and tested Hadoop. I can explore HDFS and run MapReduce example. However, when I tried to install Spark, I can't run it ...
Alexey Markov's user avatar
0 votes
0 answers
74 views

HDFS + Taking long time to copy files from hdfs to local folder

we have Hadoop cluster that used HDFS filesystem the problem is about copying file from HDFS to local folder that take huge time for example usually copy file with size of 3G takes few second , but ...
King David's user avatar
0 votes
0 answers
40 views

ZooKeeper error ; unrecognized host name for local configuration

I am using Kylin 4+ and want to use Windows and run it locally (without Hadoop). I follow this tut in their website which states that zookeeper config must be set to local like so: kylin.env.zookeeper-...
Martin Moore's user avatar
0 votes
1 answer
101 views

Does VM machine can replace physical machine,

We have 254 Physical servers when all machines are DELL servers R740. servers are part of Hadoop cluster. most of them are holding HDFS filesystem and data node & node manager services, part of ...
King David's user avatar
0 votes
0 answers
33 views

How to force Hadoop Daemon or JVM to use given hostname instead of nodes actual hostname

0 I have 5 nodes hadoop cluster with different fqdns with domain xyz.com like node1.xyz.com, node2.xyz.com ... node5.xyz.com, its hostnames are configured with this domains, so if we write hostname ...
uds0128's user avatar
0 votes
1 answer
372 views

Clear RAM Memory Cache and buffer on production Hadoop cluster with HDFS filesystem

we have Hadoop cluster with 265 Linux RHEL machines. from total 265 machines, we have 230 data nodes machines with HDFS filesystem. total memory on each data-node is 128G and we run many spark ...
King David's user avatar
1 vote
0 answers
169 views

HDP cluster + journal nodes get out of Sync

we have HDP cluster version 2.6.5 when we look on name-node logs we can see the following warning 2023-02-20 15:56:37,731 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(...
King David's user avatar
1 vote
0 answers
35 views

YARN + how to debug wget

we are testing with wget VIA port 8088 the connection from ResourceManager02 to ResourceManager01 both Resource Managers are part of YARN service , and each resource manager service installed on RHEL ...
King David's user avatar
0 votes
1 answer
907 views

Hadoop datanodes Using "{Hostname}/{IP address}:9000" to try to connect to nameNode

I have a cluster of Pis that I'm using to experiment with Hadoop. masternode is set to .190, p1 to 191 ... p4 to 194. All nodes are up and running. start-dfs.sh, stop-all.sh, etc from the master ...
Snap E Tom's user avatar
-2 votes
1 answer
128 views

How does placing data in various racks help to exploit the fact that intra-rack aggregated bandwidth>=inter-rack bandwidth?

GFS research paper snapshot it says that(my interpretation after reading research paper and its reviews) "inter rack bandwidth is lower than aggregated intra rack bandwidth(not sure what it means ...
gibmegucci's user avatar
-1 votes
2 answers
237 views

RHEL + can we improve disks performance by tuning kernel parameters?

we have Hadoop cluster and we are collection metrics collection data in order to investigate slowness behavior on spark applications after long investigation on our Hadoop cluster we noticed from ...
King David's user avatar
0 votes
1 answer
216 views

How to add multiple hostnames in private DNS zone in Azure to resolve hostnames for VNET?

I have an AKS (Azure Kubernetes cluster) that is on a VNET (Azure Virtual Network) that needs to connect to multiple On-prem hadoop machines to read/write data. I have a private DNS zone connected to ...
Venkatesh Gotimukul's user avatar
1 vote
0 answers
11 views

How to debug policy enforcement on YARN queues?

I have a HDP 3.1 cluster and it seems that the fair policy isn't behaving as expected or YARN is misconfigured, since some users/applications/jobs are consuming more resources than we supposed it to ...
jguilhermemv's user avatar
0 votes
1 answer
109 views

Hadoop recommissioning datanode

Do I need to delete all data from a datanode before recommissioning it, or it doesn't matter and the namenode will not pick stale data from the datanode?
Guido Aulisi's user avatar
2 votes
0 answers
392 views

Unable to start Apache Kylin

Good morning, I am trying to install Kylin-3.1.1 on a remote linux server, I made sure it had all software requirments and I installed already the following programms: apache-hive-3.1.2-bin , kylin-3....
user617409's user avatar
1 vote
0 answers
72 views

Backup and Restore strategy in Hbase cluster

I have just started with Hbase cluster. I have a Hbase cluster with 2 master nodes and 4 slave nodes. I have one hbase table where huge data is populated everyday so the disk gets filled quickly. I ...
Juvenik's user avatar
  • 111
0 votes
1 answer
718 views

what's the meaning of Requested resource=<memory:-1, vCores:1> in hadoop web ui?

I want to insert some data in to hive table. but it is stuck. So I go to the hadoop web ui and find the following information: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: ...
appleyuchi's user avatar
0 votes
1 answer
277 views

Change HDFS replication factor

I've changed replication factor from 3 to 2 for some directories with command: hdfs dfs -setrep -R 2 /path/to/dir but my HDFS free space still the same. Should I do something else to free my disks?
John Brown's user avatar
0 votes
1 answer
93 views

HDFS. How to free 1 particular disk

I have cluster with 3 servers. 2 of them have 2 TB disks and another one have 500 Gb SSD. I am trying to use balancer, but I still get 70% of usage on 2TB disks and 99% on 500Gb due to non-dfs files. ...
John Brown's user avatar
0 votes
1 answer
222 views

Hadoop Cluster Capacity Planning of Data Nodes for disks per data node

we are planing to build hadoop cluster with 12 data nodes machines when the replication factor is 3 and DataNode failed disk tolerance - 1 data nodes machines are include the disks for HDFS since we ...
King David's user avatar
0 votes
1 answer
2k views

What is the default password of hive

With local console, typing "hive" launching the console directly without any password. But, when I try to connect using dbeaver/beeline, it prompts for username/password. I tried with hive/&...
Uday Kiran Reddy's user avatar
0 votes
1 answer
262 views

Optimal RAID configuration for EC2 instance store used for HDFS

I'm trying to determine if there is any practical advantage to configuring a RAID array on the instance store of a 3x d2.2xlarge instances being used for HDFS. Initially I planned to just mount each ...
John R's user avatar
  • 393
1 vote
2 answers
2k views

NoNode for HBase master pseudodistributed mode

I am using Ubuntu 18.04, hadoop 3.1.3 and hbase 2.2.1 To me it seems like my hadoop and HBase are not configured correctly to interact. When I through the HBase shell try to create a table it yields ...
Jonas Grønbek's user avatar
0 votes
1 answer
20 views

Users for Hadoop deployment

I followed instructions found online to install and configuring a "2 name nodes" + "10 data nodes" hadoop cluster on CentOS 8. I created a wheel user called "hadoop" on all nodes and setup ...
Root Loop's user avatar
  • 912
1 vote
1 answer
1k views

List all files in hdfs directory

Due to some error at one component, files in HDFS got accumulated and the number is huge i.e 2123516. I want to list all files and want to copy their name in one file but when I run the following ...
innervoice's user avatar
0 votes
1 answer
966 views

mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk

We need to create xfs file-system on kafka disk The special thing about kafka disk is the disk size kafka disk have 20TB size in our case I not sure about the following mkfs , but I need advice to ...
shalom's user avatar
  • 501
0 votes
1 answer
224 views

is it possible mix different RHEL OS version in hadoop cluster?

we are using the following HDP cluster with ambari , list of nodes and their RHEL version 3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2 312 DATA-NODES machines ...
shalom's user avatar
  • 501
0 votes
0 answers
220 views

Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?

I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
HUSMEN's user avatar
  • 101
0 votes
0 answers
39 views

Request Time Out / Sessions Stalling through IPTABLE (DNAT)

Scenario: Customer recently Migrated Clustered HANA DB Servers to Azure Cloud Platform but these are Physical Servers on Azure (Offering: Azure HLI). Usually these HLIs (HANA DB Servers) in Azure ...
Ram Too's user avatar
  • 11
0 votes
1 answer
88 views

transferring data between two hadoop clusters without direct network connectivity

I have a need to transfer data fairly regularly (on demand, not scripted / streamed) between two independent hadoop clusters. One of which is deployed in an isolated network and has no direct access ...
vdrandom's user avatar
2 votes
0 answers
3k views

Yarn error: Failed to create Spark client for Spark session

I'm a bit new to this and have little experience, would appreciate your help. I'm trying to install Hive on an existing Spark installation. I mostly followed the instructions in this page with no ...
hnagaty's user avatar
  • 131
0 votes
1 answer
308 views

hadoop + can we install zookeeper servers on kafka hosts

we want to dedicated the zookeeper servers only for kafka machines so each kafka machine include the zookeeper server and zookeeper server will serve only the kafka host and not other application in ...
shalom's user avatar
  • 501
0 votes
2 answers
2k views

High Active(file) Memory Usage in Oracle Linux VMs

I recently searched and read lots of posts and questions about Linux memory management but I can't find my case. For example, there is a question in Unix StackExchange about High memory usage but no ...
Mahdizade's user avatar
1 vote
0 answers
39 views

How to install hadoop reusable on a local network?

For a multi-node image cluster on an institute, we have sevaral laptops and machines and we want to create a hadoop cluster with hbase on the top for indexing the data/images. I have tried some VMare ...
madik_atma's user avatar
0 votes
1 answer
719 views

HDFS balancing , how to balanced hdfs data?

we have Hadoop version - 2.6.4 On the datanode machine we can see that hdfs data isn’t balanced On some disks we have different used size as sdb 11G and sdd 17G /dev/sdd 20G 3.0G 17G 15% /grid/...
shalom's user avatar
  • 501
1 vote
1 answer
289 views

Hadoop: Failed to start backup node, bad state: DROP_UNTIL_NEXT_ROLL

I have created a small Hadoop cluster setup with 1 NameNode and 1 DataNode to get hands-on. below is my configuration files: Core-site.xml <property> <name>fs.defaultFS</name> ...
Dipak's user avatar
  • 111
0 votes
1 answer
203 views

Install Nvidia Drivers 9.0 for TensorFlow pip (Debian 9.7)

I installed Nvidia drivers 9.1 on my Debian 9.7 (Dataproc) when I try to run TensorFlow 1.9 via this test script it fails: Used this guide to install GPU Drivers: https://cloud.google.com/dataproc/...
gogasca's user avatar
  • 343
1 vote
1 answer
383 views

Zookeeper best practice - using only SSD disks

we have HDP clusters version 2.6.0 / 2.6.1 ( hortonworks ) , and maybe in the future the version 3.0 I searched a lot in HORTONWORKS documentation , but not found the details about using SSD disks ...
shalom's user avatar
  • 501
0 votes
1 answer
332 views

How to configure Kerberos authentication on the browsers which are on CITRIX page?

We are connecting to our secure client network via CITRIX. We are using chrome to open all quick links. like ambari etc. They open and we are good there, but other useful links like RM and HISTORY ...
akash sharma's user avatar
0 votes
0 answers
96 views

Avoid kafka disk to became 100% used by Cron job

We want to suggest the following based on our issues on kafka disks We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 ) Each cluster include 3 kafka machines , ...
shalom's user avatar
  • 501
0 votes
1 answer
64 views

Free Account Azure version HDInsight and cores issue

I am using an Free Azure account version and I am trying to create the resources needed to put in place HDInsight. I have done it twice, but in order to spare the time/money I have available, I have ...
Nicola's user avatar
  • 1
1 vote
2 answers
111 views

can we mix MTU values in cluster

we have hadoop cluster ( all machines are linux redhat machines version 7.x ) on the VM machines we set MTU=8900 and all other machines we set MTU=9000 we set on VM MTU=8900 because we saw some ...
shalom's user avatar
  • 501
0 votes
2 answers
3k views

IP Port same for source and destination

Yesterday, while debugging in a hadoop cluster i noticed something strange # netstat -taupen | grep 54310 tcp 0 0 10.0.12.209:54310 10.0.12.209:54310 TIME_WAIT You can notice that source ...
pradeepchhetri's user avatar
0 votes
1 answer
410 views

Using same password for kerberos and openldap

We have a working structure for our hadop where openldap was used for authentication with below structure,along with ranger and knox. openldap root:- dn: dc=abchadoop,dc=com,dc=za Subtree inside ...
anwaar_hell's user avatar
1 vote
0 answers
101 views

Sample output of Rumen or Input to Gridmix

I want to see JobHistory logs, which can be fed as input to the Rumen. More specifically, I am interested in knowing input format for the Gridmix. I tried following two things for it: 1) I found ...
PHcoDer's user avatar
  • 111
1 vote
1 answer
2k views

Using posix attributes instead of normal LDAP?

Due to the way a software we use interacts with Unix, when I am setting up a certain application to interact with LDAP I need to use Posix attributes instead of normal LDAP attributes. So far all I ...
Josh's user avatar
  • 111
0 votes
1 answer
39 views

Bare metal to Big Data: Can all of these operate together on the same cluster? [duplicate]

I am a VERY new sysadmin (Class of '16) and I've been asked to create a big data cluster with 3 bare metal PowerEdge Servers. I have the following request to be put on the cluster: *Hadoop2 *YARN *...
Beth L's user avatar
  • 3

1
2 3 4 5 6