Questions tagged [hadoop]
Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.
262
questions
0
votes
0
answers
42
views
HDFS + Taking long time to copy files from hdfs to local folder
we have Hadoop cluster that used HDFS filesystem
the problem is about copying file from HDFS to local folder that take huge time
for example usually copy file with size of 3G takes few second , but ...
- 559
0
votes
0
answers
30
views
ZooKeeper error ; unrecognized host name for local configuration
I am using Kylin 4+ and want to use Windows and run it locally (without Hadoop).
I follow this tut in their website which states that zookeeper config must be set to local like so:
kylin.env.zookeeper-...
0
votes
1
answer
94
views
Does VM machine can replace physical machine,
We have 254 Physical servers when all machines are DELL servers R740.
servers are part of Hadoop cluster. most of them are holding HDFS filesystem and data node & node manager services, part of ...
- 559
0
votes
0
answers
29
views
How to force Hadoop Daemon or JVM to use given hostname instead of nodes actual hostname
0
I have 5 nodes hadoop cluster with different fqdns with domain xyz.com like node1.xyz.com, node2.xyz.com ... node5.xyz.com, its hostnames are configured with this domains, so if we write hostname ...
- 1
0
votes
1
answer
235
views
Clear RAM Memory Cache and buffer on production Hadoop cluster with HDFS filesystem
we have Hadoop cluster with 265 Linux RHEL machines.
from total 265 machines, we have 230 data nodes machines with HDFS filesystem.
total memory on each data-node is 128G and we run many spark ...
- 559
1
vote
0
answers
135
views
HDP cluster + journal nodes get out of Sync
we have HDP cluster version 2.6.5
when we look on name-node logs we can see the following warning
2023-02-20 15:56:37,731 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(...
- 559
1
vote
0
answers
34
views
YARN + how to debug wget
we are testing with wget VIA port 8088 the connection from ResourceManager02 to ResourceManager01
both Resource Managers are part of YARN service , and each resource manager service installed on RHEL ...
- 559
0
votes
1
answer
704
views
Hadoop datanodes Using "{Hostname}/{IP address}:9000" to try to connect to nameNode
I have a cluster of Pis that I'm using to experiment with Hadoop. masternode is set to .190, p1 to 191 ... p4 to 194. All nodes are up and running. start-dfs.sh, stop-all.sh, etc from the master ...
- 101
-2
votes
1
answer
101
views
How does placing data in various racks help to exploit the fact that intra-rack aggregated bandwidth>=inter-rack bandwidth?
GFS research paper snapshot
it says that(my interpretation after reading research paper and its reviews) "inter rack bandwidth is lower than aggregated intra rack bandwidth(not sure what it means ...
-1
votes
2
answers
180
views
RHEL + can we improve disks performance by tuning kernel parameters?
we have Hadoop cluster and we are collection metrics collection data in order to investigate slowness behavior on spark applications
after long investigation on our Hadoop cluster
we noticed from ...
- 559
0
votes
1
answer
186
views
How to add multiple hostnames in private DNS zone in Azure to resolve hostnames for VNET?
I have an AKS (Azure Kubernetes cluster) that is on a VNET (Azure Virtual Network) that needs to connect to multiple On-prem hadoop machines to read/write data. I have a private DNS zone connected to ...
1
vote
0
answers
11
views
How to debug policy enforcement on YARN queues?
I have a HDP 3.1 cluster and it seems that the fair policy isn't behaving as expected or YARN is misconfigured, since some users/applications/jobs are consuming more resources than we supposed it to ...
- 111
0
votes
1
answer
99
views
Hadoop recommissioning datanode
Do I need to delete all data from a datanode before recommissioning it, or it doesn't matter and the namenode will not pick stale data from the datanode?
2
votes
0
answers
361
views
Unable to start Apache Kylin
Good morning, I am trying to install Kylin-3.1.1 on a remote linux server, I made sure it had all software requirments and I installed already the following programms: apache-hive-3.1.2-bin , kylin-3....
- 21
1
vote
0
answers
60
views
Backup and Restore strategy in Hbase cluster
I have just started with Hbase cluster. I have a Hbase cluster with 2 master nodes and 4 slave nodes. I have one hbase table where huge data is populated everyday so the disk gets filled quickly. I ...
- 111
0
votes
1
answer
668
views
what's the meaning of Requested resource=<memory:-1, vCores:1> in hadoop web ui?
I want to insert some data in to hive table.
but it is stuck.
So I go to the hadoop web ui and find the following information:
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: ...
0
votes
1
answer
232
views
Change HDFS replication factor
I've changed replication factor from 3 to 2 for some directories with command:
hdfs dfs -setrep -R 2 /path/to/dir
but my HDFS free space still the same. Should I do something else to free my disks?
0
votes
1
answer
77
views
HDFS. How to free 1 particular disk
I have cluster with 3 servers. 2 of them have 2 TB disks and another one have 500 Gb SSD. I am trying to use balancer, but I still get 70% of usage on 2TB disks and 99% on 500Gb due to non-dfs files. ...
0
votes
1
answer
216
views
Hadoop Cluster Capacity Planning of Data Nodes for disks per data node
we are planing to build hadoop cluster with 12 data nodes machines
when the replication factor is 3
and DataNode failed disk tolerance - 1
data nodes machines are include the disks for HDFS
since we ...
- 559
0
votes
1
answer
2k
views
What is the default password of hive
With local console, typing "hive" launching the console directly without any password.
But, when I try to connect using dbeaver/beeline, it prompts for username/password.
I tried with
hive/&...
- 129
0
votes
1
answer
259
views
Optimal RAID configuration for EC2 instance store used for HDFS
I'm trying to determine if there is any practical advantage to configuring a RAID array on the instance store of a 3x d2.2xlarge instances being used for HDFS. Initially I planned to just mount each ...
- 383
1
vote
2
answers
2k
views
NoNode for HBase master pseudodistributed mode
I am using Ubuntu 18.04, hadoop 3.1.3 and hbase 2.2.1
To me it seems like my hadoop and HBase are not configured correctly to interact. When I through the HBase shell try to create a table it yields ...
- 123
0
votes
1
answer
20
views
Users for Hadoop deployment
I followed instructions found online to install and configuring a "2 name nodes" + "10 data nodes" hadoop cluster on CentOS 8.
I created a wheel user called "hadoop" on all nodes and setup ...
- 902
1
vote
1
answer
1k
views
List all files in hdfs directory
Due to some error at one component, files in HDFS got accumulated and the number is huge i.e 2123516. I want to list all files and want to copy their name in one file but when I run the following ...
- 21
0
votes
1
answer
858
views
mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk
We need to create xfs file-system on kafka disk
The special thing about kafka disk is the disk size
kafka disk have 20TB size in our case
I not sure about the following mkfs , but I need advice to ...
- 471
0
votes
1
answer
218
views
is it possible mix different RHEL OS version in hadoop cluster?
we are using the following HDP cluster with ambari ,
list of nodes and their RHEL version
3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2
312 DATA-NODES machines ...
- 471
0
votes
0
answers
214
views
Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?
I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
- 1
0
votes
0
answers
39
views
Request Time Out / Sessions Stalling through IPTABLE (DNAT)
Scenario:
Customer recently Migrated Clustered HANA DB Servers to Azure Cloud Platform but these are Physical Servers on Azure (Offering: Azure HLI). Usually these HLIs (HANA DB Servers) in Azure ...
- 11
0
votes
1
answer
77
views
transferring data between two hadoop clusters without direct network connectivity
I have a need to transfer data fairly regularly (on demand, not scripted / streamed) between two independent hadoop clusters. One of which is deployed in an isolated network and has no direct access ...
- 1
2
votes
0
answers
3k
views
Yarn error: Failed to create Spark client for Spark session
I'm a bit new to this and have little experience, would appreciate your help.
I'm trying to install Hive on an existing Spark installation.
I mostly followed the instructions in this page with no ...
- 131
0
votes
1
answer
308
views
hadoop + can we install zookeeper servers on kafka hosts
we want to dedicated the zookeeper servers only for kafka machines
so each kafka machine include the zookeeper server
and zookeeper server will serve only the kafka host and not other application
in ...
- 471
0
votes
2
answers
2k
views
High Active(file) Memory Usage in Oracle Linux VMs
I recently searched and read lots of posts and questions about Linux memory management but I can't find my case. For example, there is a question in Unix StackExchange about High memory usage but no ...
1
vote
0
answers
39
views
How to install hadoop reusable on a local network?
For a multi-node image cluster on an institute, we have sevaral laptops and machines and we want to create a hadoop cluster with hbase on the top for indexing the data/images.
I have tried some VMare ...
- 111
0
votes
1
answer
703
views
HDFS balancing , how to balanced hdfs data?
we have Hadoop version - 2.6.4
On the datanode machine we can see that hdfs data isn’t balanced
On some disks we have different used size as sdb 11G and sdd 17G
/dev/sdd 20G 3.0G 17G 15% /grid/...
- 471
1
vote
1
answer
274
views
Hadoop: Failed to start backup node, bad state: DROP_UNTIL_NEXT_ROLL
I have created a small Hadoop cluster setup with 1 NameNode and 1 DataNode to get hands-on.
below is my configuration files:
Core-site.xml
<property>
<name>fs.defaultFS</name>
...
- 111
0
votes
1
answer
203
views
Install Nvidia Drivers 9.0 for TensorFlow pip (Debian 9.7)
I installed Nvidia drivers 9.1 on my Debian 9.7 (Dataproc)
when I try to run TensorFlow 1.9 via this test script it fails:
Used this guide to install GPU Drivers: https://cloud.google.com/dataproc/...
google-cloud-platform
1
vote
1
answer
368
views
Zookeeper best practice - using only SSD disks
we have HDP clusters version 2.6.0 / 2.6.1 ( hortonworks ) , and maybe in the future the version 3.0
I searched a lot in HORTONWORKS documentation , but not found the details about using SSD disks ...
- 471
0
votes
1
answer
293
views
How to configure Kerberos authentication on the browsers which are on CITRIX page?
We are connecting to our secure client network via CITRIX. We are using chrome to open all quick links. like ambari etc. They open and we are good there, but other useful links like RM and HISTORY ...
- 103
0
votes
0
answers
93
views
Avoid kafka disk to became 100% used by Cron job
We want to suggest the following based on our issues on kafka disks
We have many HDP clusters ( based on ambari , and all machines are redhat version 7.2 )
Each cluster include 3 kafka machines , ...
- 471
0
votes
1
answer
58
views
Free Account Azure version HDInsight and cores issue
I am using an Free Azure account version and I am trying to create the resources needed to put in place HDInsight. I have done it twice, but in order to spare the time/money I have available, I have ...
- 1
1
vote
2
answers
102
views
can we mix MTU values in cluster
we have hadoop cluster ( all machines are linux redhat machines version 7.x )
on the VM machines we set MTU=8900 and all other machines we set MTU=9000
we set on VM MTU=8900 because we saw some ...
- 471
0
votes
2
answers
3k
views
IP Port same for source and destination
Yesterday, while debugging in a hadoop cluster i noticed something strange
# netstat -taupen | grep 54310
tcp 0 0 10.0.12.209:54310 10.0.12.209:54310 TIME_WAIT
You can notice that source ...
- 2,788
0
votes
1
answer
387
views
Using same password for kerberos and openldap
We have a working structure for our hadop where openldap was used for authentication with below structure,along with ranger and knox.
openldap root:-
dn: dc=abchadoop,dc=com,dc=za
Subtree inside ...
- 101
1
vote
0
answers
101
views
Sample output of Rumen or Input to Gridmix
I want to see JobHistory logs, which can be fed as input to the Rumen. More specifically, I am interested in knowing input format for the Gridmix.
I tried following two things for it:
1) I found ...
- 111
1
vote
1
answer
2k
views
Using posix attributes instead of normal LDAP?
Due to the way a software we use interacts with Unix, when I am setting up a certain application to interact with LDAP I need to use Posix attributes instead of normal LDAP attributes.
So far all I ...
- 111
0
votes
1
answer
39
views
Bare metal to Big Data: Can all of these operate together on the same cluster? [duplicate]
I am a VERY new sysadmin (Class of '16) and I've been asked to create a big data cluster with 3 bare metal PowerEdge Servers. I have the following request to be put on the cluster:
*Hadoop2 *YARN *...
- 3
0
votes
0
answers
720
views
Using OpenLDAP as back end for Kerberos
We want to integrate our existing security setup (Apache Knox, OpenLDAP, Apache Ranger) with Kerberos. So what I understood through some blogs that we can use OpenLDAP as a back-end for the Kerberos ...
- 101
0
votes
1
answer
387
views
dose setting Jumbo frame on linux redhat influencing OS performance?
We set the MTU to 9000 on all our linux machines ( we have redhat machines version - 7.3 ) , linux machines are part of hadoop clusters
we want to know if set the MTU to 9000 can be negative on OS ...
- 471
0
votes
1
answer
4k
views
how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster
we have ambari cluster ( version 2.6 ) with 3 workers machine , and each worker machine have 16 CPU CORE ( see pic down ) , while each machine have 32G memory
according to:
yarn.nodemanager.resource....
- 471
1
vote
0
answers
61
views
spark.dynamicAllocation + setting the spark parameters according to ambari cluster
we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,,
spark....
- 471