Questions tagged [hadoop]
Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.
266
questions
0
votes
0
answers
55
views
YARN resource manager "HTTP request sent, awaiting response..."
we have Hadoop cluster with active/stand by resource manager services the active resource manager is on master1 machine and the stand by resource manager is on master2 machine
in our cluster YARN ...
0
votes
0
answers
31
views
PostgreSQL + Removing old Host from DB not succeeded
We have HDP cluster with 152 workers machines - worker1.duplex.com .. worker152.duplex.com , While all machines are installed on RHEL 7.9 version
We are trying to delete the last host - worker152....
0
votes
0
answers
30
views
Hadoop Namenode heap size tuning
NameNode process is executed in Java virtual machine,
and Java object which NameNode creates is managed in Java virtual memory. As the fles or directories are created, inode objects and block objects ...
0
votes
0
answers
20
views
EOF Exception whnen runnign spark-shell on Hadoop Cluster
The problem
I am trying to install Spark on Hadoop cluster. I've installed and tested Hadoop. I can explore HDFS and run MapReduce example. However, when I tried to install Spark, I can't run it ...
0
votes
0
answers
74
views
HDFS + Taking long time to copy files from hdfs to local folder
we have Hadoop cluster that used HDFS filesystem
the problem is about copying file from HDFS to local folder that take huge time
for example usually copy file with size of 3G takes few second , but ...
0
votes
0
answers
40
views
ZooKeeper error ; unrecognized host name for local configuration
I am using Kylin 4+ and want to use Windows and run it locally (without Hadoop).
I follow this tut in their website which states that zookeeper config must be set to local like so:
kylin.env.zookeeper-...
0
votes
1
answer
101
views
Does VM machine can replace physical machine,
We have 254 Physical servers when all machines are DELL servers R740.
servers are part of Hadoop cluster. most of them are holding HDFS filesystem and data node & node manager services, part of ...
0
votes
0
answers
33
views
How to force Hadoop Daemon or JVM to use given hostname instead of nodes actual hostname
0
I have 5 nodes hadoop cluster with different fqdns with domain xyz.com like node1.xyz.com, node2.xyz.com ... node5.xyz.com, its hostnames are configured with this domains, so if we write hostname ...
0
votes
1
answer
372
views
Clear RAM Memory Cache and buffer on production Hadoop cluster with HDFS filesystem
we have Hadoop cluster with 265 Linux RHEL machines.
from total 265 machines, we have 230 data nodes machines with HDFS filesystem.
total memory on each data-node is 128G and we run many spark ...
1
vote
0
answers
169
views
HDP cluster + journal nodes get out of Sync
we have HDP cluster version 2.6.5
when we look on name-node logs we can see the following warning
2023-02-20 15:56:37,731 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(...
1
vote
0
answers
35
views
YARN + how to debug wget
we are testing with wget VIA port 8088 the connection from ResourceManager02 to ResourceManager01
both Resource Managers are part of YARN service , and each resource manager service installed on RHEL ...
0
votes
1
answer
907
views
Hadoop datanodes Using "{Hostname}/{IP address}:9000" to try to connect to nameNode
I have a cluster of Pis that I'm using to experiment with Hadoop. masternode is set to .190, p1 to 191 ... p4 to 194. All nodes are up and running. start-dfs.sh, stop-all.sh, etc from the master ...
-2
votes
1
answer
128
views
How does placing data in various racks help to exploit the fact that intra-rack aggregated bandwidth>=inter-rack bandwidth?
GFS research paper snapshot
it says that(my interpretation after reading research paper and its reviews) "inter rack bandwidth is lower than aggregated intra rack bandwidth(not sure what it means ...
-1
votes
2
answers
237
views
RHEL + can we improve disks performance by tuning kernel parameters?
we have Hadoop cluster and we are collection metrics collection data in order to investigate slowness behavior on spark applications
after long investigation on our Hadoop cluster
we noticed from ...
0
votes
1
answer
216
views
How to add multiple hostnames in private DNS zone in Azure to resolve hostnames for VNET?
I have an AKS (Azure Kubernetes cluster) that is on a VNET (Azure Virtual Network) that needs to connect to multiple On-prem hadoop machines to read/write data. I have a private DNS zone connected to ...
1
vote
0
answers
11
views
How to debug policy enforcement on YARN queues?
I have a HDP 3.1 cluster and it seems that the fair policy isn't behaving as expected or YARN is misconfigured, since some users/applications/jobs are consuming more resources than we supposed it to ...
0
votes
1
answer
109
views
Hadoop recommissioning datanode
Do I need to delete all data from a datanode before recommissioning it, or it doesn't matter and the namenode will not pick stale data from the datanode?
2
votes
0
answers
392
views
Unable to start Apache Kylin
Good morning, I am trying to install Kylin-3.1.1 on a remote linux server, I made sure it had all software requirments and I installed already the following programms: apache-hive-3.1.2-bin , kylin-3....
1
vote
0
answers
72
views
Backup and Restore strategy in Hbase cluster
I have just started with Hbase cluster. I have a Hbase cluster with 2 master nodes and 4 slave nodes. I have one hbase table where huge data is populated everyday so the disk gets filled quickly. I ...
0
votes
1
answer
718
views
what's the meaning of Requested resource=<memory:-1, vCores:1> in hadoop web ui?
I want to insert some data in to hive table.
but it is stuck.
So I go to the hadoop web ui and find the following information:
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: ...
0
votes
1
answer
277
views
Change HDFS replication factor
I've changed replication factor from 3 to 2 for some directories with command:
hdfs dfs -setrep -R 2 /path/to/dir
but my HDFS free space still the same. Should I do something else to free my disks?
0
votes
1
answer
93
views
HDFS. How to free 1 particular disk
I have cluster with 3 servers. 2 of them have 2 TB disks and another one have 500 Gb SSD. I am trying to use balancer, but I still get 70% of usage on 2TB disks and 99% on 500Gb due to non-dfs files. ...
0
votes
1
answer
222
views
Hadoop Cluster Capacity Planning of Data Nodes for disks per data node
we are planing to build hadoop cluster with 12 data nodes machines
when the replication factor is 3
and DataNode failed disk tolerance - 1
data nodes machines are include the disks for HDFS
since we ...
0
votes
1
answer
2k
views
What is the default password of hive
With local console, typing "hive" launching the console directly without any password.
But, when I try to connect using dbeaver/beeline, it prompts for username/password.
I tried with
hive/&...
0
votes
1
answer
262
views
Optimal RAID configuration for EC2 instance store used for HDFS
I'm trying to determine if there is any practical advantage to configuring a RAID array on the instance store of a 3x d2.2xlarge instances being used for HDFS. Initially I planned to just mount each ...
1
vote
2
answers
2k
views
NoNode for HBase master pseudodistributed mode
I am using Ubuntu 18.04, hadoop 3.1.3 and hbase 2.2.1
To me it seems like my hadoop and HBase are not configured correctly to interact. When I through the HBase shell try to create a table it yields ...
0
votes
1
answer
20
views
Users for Hadoop deployment
I followed instructions found online to install and configuring a "2 name nodes" + "10 data nodes" hadoop cluster on CentOS 8.
I created a wheel user called "hadoop" on all nodes and setup ...
1
vote
1
answer
1k
views
List all files in hdfs directory
Due to some error at one component, files in HDFS got accumulated and the number is huge i.e 2123516. I want to list all files and want to copy their name in one file but when I run the following ...
0
votes
1
answer
966
views
mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk
We need to create xfs file-system on kafka disk
The special thing about kafka disk is the disk size
kafka disk have 20TB size in our case
I not sure about the following mkfs , but I need advice to ...
0
votes
1
answer
224
views
is it possible mix different RHEL OS version in hadoop cluster?
we are using the following HDP cluster with ambari ,
list of nodes and their RHEL version
3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2
312 DATA-NODES machines ...
0
votes
0
answers
220
views
Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?
I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
0
votes
0
answers
39
views
Request Time Out / Sessions Stalling through IPTABLE (DNAT)
Scenario:
Customer recently Migrated Clustered HANA DB Servers to Azure Cloud Platform but these are Physical Servers on Azure (Offering: Azure HLI). Usually these HLIs (HANA DB Servers) in Azure ...
0
votes
1
answer
88
views
transferring data between two hadoop clusters without direct network connectivity
I have a need to transfer data fairly regularly (on demand, not scripted / streamed) between two independent hadoop clusters. One of which is deployed in an isolated network and has no direct access ...
2
votes
0
answers
3k
views
Yarn error: Failed to create Spark client for Spark session
I'm a bit new to this and have little experience, would appreciate your help.
I'm trying to install Hive on an existing Spark installation.
I mostly followed the instructions in this page with no ...
0
votes
1
answer
308
views
hadoop + can we install zookeeper servers on kafka hosts
we want to dedicated the zookeeper servers only for kafka machines
so each kafka machine include the zookeeper server
and zookeeper server will serve only the kafka host and not other application
in ...
0
votes
2
answers
2k
views
High Active(file) Memory Usage in Oracle Linux VMs
I recently searched and read lots of posts and questions about Linux memory management but I can't find my case. For example, there is a question in Unix StackExchange about High memory usage but no ...
1
vote
0
answers
39
views
How to install hadoop reusable on a local network?
For a multi-node image cluster on an institute, we have sevaral laptops and machines and we want to create a hadoop cluster with hbase on the top for indexing the data/images.
I have tried some VMare ...
0
votes
1
answer
719
views
HDFS balancing , how to balanced hdfs data?
we have Hadoop version - 2.6.4
On the datanode machine we can see that hdfs data isn’t balanced
On some disks we have different used size as sdb 11G and sdd 17G
/dev/sdd 20G 3.0G 17G 15% /grid/...
1
vote
1
answer
289
views
Hadoop: Failed to start backup node, bad state: DROP_UNTIL_NEXT_ROLL
I have created a small Hadoop cluster setup with 1 NameNode and 1 DataNode to get hands-on.
below is my configuration files:
Core-site.xml
<property>
<name>fs.defaultFS</name>
...
0
votes
1
answer
203
views
Install Nvidia Drivers 9.0 for TensorFlow pip (Debian 9.7)
I installed Nvidia drivers 9.1 on my Debian 9.7 (Dataproc)
when I try to run TensorFlow 1.9 via this test script it fails:
Used this guide to install GPU Drivers: https://cloud.google.com/dataproc/...
1
vote
1
answer
383
views
Zookeeper best practice - using only SSD disks
we have HDP clusters version 2.6.0 / 2.6.1 ( hortonworks ) , and maybe in the future the version 3.0
I searched a lot in HORTONWORKS documentation , but not found the details about using SSD disks ...
0
votes
1
answer
332
views
How to configure Kerberos authentication on the browsers which are on CITRIX page?
We are connecting to our secure client network via CITRIX. We are using chrome to open all quick links. like ambari etc. They open and we are good there, but other useful links like RM and HISTORY ...
0
votes
0
answers
96
views
Avoid kafka disk to became 100% used by Cron job
We want to suggest the following based on our issues on kafka disks
We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 )
Each cluster include 3 kafka machines , ...
0
votes
1
answer
64
views
Free Account Azure version HDInsight and cores issue
I am using an Free Azure account version and I am trying to create the resources needed to put in place HDInsight. I have done it twice, but in order to spare the time/money I have available, I have ...
1
vote
2
answers
111
views
can we mix MTU values in cluster
we have hadoop cluster ( all machines are linux redhat machines version 7.x )
on the VM machines we set MTU=8900 and all other machines we set MTU=9000
we set on VM MTU=8900 because we saw some ...
0
votes
2
answers
3k
views
IP Port same for source and destination
Yesterday, while debugging in a hadoop cluster i noticed something strange
# netstat -taupen | grep 54310
tcp 0 0 10.0.12.209:54310 10.0.12.209:54310 TIME_WAIT
You can notice that source ...
0
votes
1
answer
410
views
Using same password for kerberos and openldap
We have a working structure for our hadop where openldap was used for authentication with below structure,along with ranger and knox.
openldap root:-
dn: dc=abchadoop,dc=com,dc=za
Subtree inside ...
1
vote
0
answers
101
views
Sample output of Rumen or Input to Gridmix
I want to see JobHistory logs, which can be fed as input to the Rumen. More specifically, I am interested in knowing input format for the Gridmix.
I tried following two things for it:
1) I found ...
1
vote
1
answer
2k
views
Using posix attributes instead of normal LDAP?
Due to the way a software we use interacts with Unix, when I am setting up a certain application to interact with LDAP I need to use Posix attributes instead of normal LDAP attributes.
So far all I ...
0
votes
1
answer
39
views
Bare metal to Big Data: Can all of these operate together on the same cluster? [duplicate]
I am a VERY new sysadmin (Class of '16) and I've been asked to create a big data cluster with 3 bare metal PowerEdge Servers. I have the following request to be put on the cluster:
*Hadoop2 *YARN *...