Questions tagged [hadoop]

Hadoop is an open-source solution for providing a distributed/replicated file system, a produciton grade map-reduce system, and has a series of complementary additions like Hive, Pig, and HBase to get more out of a Hadoop-powered cluster.

Filter by
Sorted by
Tagged with
0 votes
0 answers
754 views

Using OpenLDAP as back end for Kerberos

We want to integrate our existing security setup (Apache Knox, OpenLDAP, Apache Ranger) with Kerberos. So what I understood through some blogs that we can use OpenLDAP as a back-end for the Kerberos ...
anwaar_hell's user avatar
0 votes
1 answer
406 views

dose setting Jumbo frame on linux redhat influencing OS performance?

We set the MTU to 9000 on all our linux machines ( we have redhat machines version - 7.3 ) , linux machines are part of hadoop clusters we want to know if set the MTU to 9000 can be negative on OS ...
shalom's user avatar
  • 501
0 votes
1 answer
4k views

how to determine yarn.scheduler.maximum-allocation-vcores value in ambari cluster

we have ambari cluster ( version 2.6 ) with 3 workers machine , and each worker machine have 16 CPU CORE ( see pic down ) , while each machine have 32G memory according to: yarn.nodemanager.resource....
shalom's user avatar
  • 501
1 vote
0 answers
63 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
shalom's user avatar
  • 501
6 votes
0 answers
10k views

Spark Error: Failed to Send RPC to Datanode

We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now. From the log we can see an error message: Failed to send RPC 9053901149358924945 ...
shalom's user avatar
  • 501
1 vote
0 answers
191 views

ambari cluster + when need to set Block replication to 1

We get the following in Spark logs: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
shalom's user avatar
  • 501
0 votes
1 answer
804 views

why ambari agent insist to create another repository file

we are installing the new hadoop version - 2.6.3.0 on ambari - 2.6.0 from ambari agent log we see the follwing: Writing File['/etc/yum.repos.d/ambari-hdp-51.repo'] because contents don't match why ...
shalom's user avatar
  • 501
1 vote
1 answer
99 views

Redirecting Ambari-Server backup file creation to a different location

I am taking backup of my Ambari server using the command ambari-server backup This creates the backup file in the location /var/lib/ambari-server/ I want the backup to go to a different location, ...
Gautam Somani's user avatar
0 votes
0 answers
42 views

Datanode machines disks size

is it important that ( workers ) datanode machines disks will be with the same size? for example we have ambari cluster with 3 workers machines ( datanode machines ) each datanode machine have 10 ...
shalom's user avatar
  • 501
1 vote
0 answers
31 views

dose ambari cluster needs ssh access between ambari-server machine to all other host

We installed ambari cluster with 3 masters machines While ambari server installed on master02 linux machine ambari cluster also include 25 DataNodes machines and 5 kafka's machines dose ambari-...
shalom's user avatar
  • 501
1 vote
0 answers
95 views

CDH Community Edition Upgrade from 5.7 to 5.13 without Cloudera Express or Cloudera Enterprise

I am having a cluster like below - 3 Hbase Master (1 Active & 2 standby) 4 Region Servers 4 Data Nodes 1 Primary & 1 Secondary Name Node 3 Journal Node 4 Nodemanager 3 Resource Manager (1 ...
tuk's user avatar
  • 343
1 vote
1 answer
497 views

kafka + how to revert topic deletion

just in case we delete the wrong topic as /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper hdpmaster01:2181 --delete --topic gtom.poli.pri.proc Topic gtom.poli.pri.procis already ...
jango's user avatar
  • 69
0 votes
1 answer
2k views

what is the safe and best way to delete the kafka topic folders

on all our kafka machines ( production machines ) , we see that: ( no free space ) df -h /var/kafka Filesystem Size Used Avail Use% Mounted on /dev/sdb 11T 11T 2.3M 100% /var/kafka ...
jango's user avatar
  • 69
5 votes
1 answer
3k views

Issues Programmatically Adding ODBC DSN to ODBC Administrator on Windows

I am working on trying to automate some configuration, and as part of that we need to add an ODBC DSN through a script. The driver I'm trying to use is the Cloudera Impala ODBC Connector, downloaded ...
Dave McGinnis's user avatar
0 votes
1 answer
373 views

grafana + graphite + failed on test connection

first I'm new to Grafana so excuse me if i don't use the right words to describe my issue I installed Grafana + Graphite solution in my ambari cluster ( I am using ambari cluster with graphite ...
shalom's user avatar
  • 501
0 votes
1 answer
453 views

what is effected when running - hadoop namenode -format

we have amabri cluster ( version 2.6 ) with 24 workers machines we want to run following commands only on worker23 machine ( because problem on worker23 ) , dose these commands effected on all ...
jango's user avatar
  • 69
1 vote
0 answers
56 views

How to prevent arbitrary executable execution on hadoop cluster

I am involved with configuring a Hadoop cluster for complete auditability and security. I am new to the Hadoop ecosystem, but I have a decent idea of the basics. I have a few concerns for which I hope ...
STN's user avatar
  • 111
2 votes
2 answers
8k views

How to identify and fix Missing blocks reported by Ambari for the NameNode?

Ambari is generating an alert NameNode Blocks Health: Total Blocks:[38252543], Missing Blocks:[2]. No further information. I've run hdfs fsck / which is reporting the entire filesystem as healthy. I'...
Michael Martinez's user avatar
0 votes
1 answer
3k views

calc md5 checksum for all files in hadoop directory

iam using hadoop apache 2.7.1 on centos and iam new to centos if i want to calc md5 checksum for specific file in hadoop i can issue the following command hdfs dfs -cat /hadoophome/myfile | ...
oula alshiekh's user avatar
1 vote
0 answers
403 views

YARN AM logs report different time-stamp from what is shown in terminal and sparkscala shell

I am trying to understand why the following occurred: I have a Docker container with Yarn and Spark running fine except that the timestamp of that container was minus X hours of what I wanted it to ...
rudimuse's user avatar
-1 votes
1 answer
4k views

Unable to ssh localhost without password despite proper perms, key in authorized_keys

I have a key ~/.ssh/id_rsa and I added the pub key to my authorized keys: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys I also changed my permissions to 600: sudo chmod 600 ~/.ssh/...
Don Rhummy's user avatar
0 votes
1 answer
489 views

How to reconfigure Ambari services values with blueprint.json file

we have many Ambari LAB clusters - Apache Ambari Version 2.5.0.3 , while ambari agent installed on Linux redhat machines my target is to find a way to update the values of services , on all the ...
shalom's user avatar
  • 501
0 votes
1 answer
393 views

Hadoop Cluster Hardware: Few large vs many small

we are looking for some help deciding what hardware to buy to support an internal Hadoop cluster. My company currently uses 1 dedicated server for Hadoop which has 196GB ram and 24cores and 6 1TB SATA ...
TWith2Sugars's user avatar
0 votes
1 answer
2k views

Apache Ambari and how to check Ambari services status

How to verify service statuses via Ambari? For example, I want to check in Ambari if the HDFS service is stopped or has been STARTED. Until now I use the following syntax in order to check the ...
jango's user avatar
  • 69
0 votes
1 answer
440 views

zookeeper automatic start operation fails when firewall is stoped at rc.local script

iam using hadoop apache 2.7.1 on centos 7 and my cluster is ha cluster and iam using zookeeper quorum for automatic failover but i want to automate zookeeper start process and ofcourse in the shell ...
oula alshiekh's user avatar
4 votes
1 answer
23k views

mount.nfs: mount system call failed

I am trying to mount hdfs on my local machine running Ubuntu using the following command :--- sudo mount -t nfs -o vers=3,proto=tcp,nolock 192.168.170.52:/ /mnt/hdfs_mount/ But I am getting this ...
Bhavya Jain's user avatar
0 votes
1 answer
1k views

copying files in hdfs stalls

Have a 35 node cluster with a high number of blocks in it: ≈450K blocks per data node. After configuration change (which contained rack reassignments and NameNode Xmx increase) HDFS became a problem. ...
inteloid's user avatar
  • 101
0 votes
2 answers
6k views

Namenodes fails starting on HA cluster - Fatals exists in Journalnode logs

I am having some problem with my Hadoop Cluster Centos 7.3 Hortonworks Ambari 2.4.2 Hortonworks HDP 2.5.3 Ambari stderr: 2017-04-06 10:49:49,039 - Getting jmx metrics from NN failed. URL: http://...
Sedat Kestepe's user avatar
0 votes
1 answer
144 views

how to install hadoop2.4.1 in windows with spark 2.0.0

i want to setup a cluster using hadoop in yarn mode..i want to use spark API for map-reduce and will use spark submit to deploy my applications..i want to work on cluster..can anyone help me how to ...
Sadim Nadeem's user avatar
3 votes
0 answers
167 views

How can I launch hdfs on Mesos without DC/OS?

From my understand DC/OS is a freemium managed service. Because I'd rather just have a raw Mesos implementation, I'd rather not be dependent on DC/OS and so I just want to know how to implement HDFS ...
Dr.Knowitall's user avatar
0 votes
0 answers
92 views

Scale Up Hadoop disk on aws infrastructure

Our Hadoop cluster disk is getting full. So I want to scale up disk asap, and one way I can think is to increase ebs volume on all three nodes Hadoop 2.6.0-cdh5.5.1 (2 data-node, 1 name-node). So my ...
piyushmandovra's user avatar
0 votes
2 answers
6k views

Try run Hive and have Error:“java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-alpha1”

I have a version of Hadoop on Ubuntu 16.10 Hadoop 3.0.0-alpha1 Source code repository https://git-wip-us.apache.org/repos/asf/hadoop.git -r a990d2ebcd6de5d7dc2d3684930759b0f0ea4dc3 Compiled by andrew ...
Nikolay Baranenko's user avatar
7 votes
1 answer
12k views

Set up a Windows 10 Client for a Linux KDC Realm

I set up a KDC Server and created a Realm EXAMPLE.COM. Here is my krb5.conf file: [libdefaults] renew_lifetime = 7d forwardable = true default_realm = EXAMPLE.COM ticket_lifetime = 24h ...
D. Müller's user avatar
0 votes
1 answer
528 views

why cant I only access a port from localhost?

I am trying to set up elasticsearch on my mapR issued redhat virtual machine. It comes pre-loaded with the mapR ecosystem. I installed elasticsearch via yum. I am able to listen to it from inside the ...
user3221132's user avatar
0 votes
1 answer
504 views

Hadoop datanode - start with one disk and add more later or start with as much disks as possible and fill them equally

I'm wondering the following in regard to the Datanode disks setup in Hadoop cluster. Which of the those two options is better: To add one (or few) disks to the Datanode, and attach more after they ...
mart's user avatar
  • 3
1 vote
0 answers
20 views

Need to set 000 permission to specific hdfs data block through commandline

I am trying to set the “000” permission to the specific block. I used below command to find the block information: su - hdfs -c "hdfs fsck -locations -files -blocks /user/rohit/...
Rohit Agrawal's user avatar
1 vote
1 answer
4k views

Hadoop FileAlreadyExistsException: Output directory hdfs://<namenode public dns>:9000/input already exists

I have Hadoop setup in fully distributed mode with one master and 3 slaves. I am trying to execute a jar file named Tasks.jar which is taking arg[0] as input directory and arg[1] as output directory. ...
Harinarayanan Mohan's user avatar
2 votes
0 answers
4k views

Hadoop Streaming with Python 3.5: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127

I'm trying to run my own mapper and reducer Python scripts using Hadoop Streaming on my cluster built on VMware Workstation VMs. Hadoop version - 2.7, Python - 3.5, OS - CentOS 7.2 on all the VMs. I ...
alex's user avatar
  • 21
2 votes
0 answers
46 views

AWS-Hadoop Data Analytics Implementation for Multiple JSON Files

I am new to hadoop and AWS. I have setup multi-node (4 instances t2.large) AWS EC2 cluster with cloudera Hadoop distribution. I have tested the environment with basic examples using CSV files such as ...
Rash's user avatar
  • 21
2 votes
1 answer
2k views

Running HDFS with only 1 data node - appending fails

I'm trying to test some services that require HDFS using Docker Compose. Since the services being tested, namenode, and data node(s) will all be running on the same physical machine (dev laptop), it ...
Robert Fraser's user avatar
1 vote
0 answers
142 views

Why Hadoop TeraSort not using all cluster nodes

Question Regarding the TeraSort demo in hadoop, please suggest if the symptom is as expected or the workload should be distributed. Symptom Started Hadoop (3 nodes in a cluster) and run the ...
mon's user avatar
  • 265
4 votes
1 answer
293 views

Possible to ssh into a server without using -i flag for key?

I have 3 EC2 instances and they all use the same private key. I'm setting up a hadoop cluster between these nodes and they require passwordless entry for this to work. How can I use this private key ...
coderkid's user avatar
  • 193
1 vote
0 answers
27 views

Is it possible to configure hdfs in a federation mode and in an HA mode in the same time?

I don't understand if it possible to configure HDFS in both modes in the same time. Does it make sense? Can somebody show a simple configuration of HDFS in both modes? (nameNode1, nameNode2, ...
Oleksandr's user avatar
  • 763
0 votes
1 answer
832 views

Why does DFSZKFailoverController kills Namenode process in hadoop?

I try to configure hadoop high availability cluster by following this tutorial: http://www.edureka.co/blog/how-to-set-up-hadoop-cluster-with-hdfs-high-availability/ When I follow that article I faces ...
Oleksandr's user avatar
  • 763
0 votes
1 answer
725 views

What should HADOOP_PREFIX be for Accumulo installation?

I'm trying to install Accumulo 1.7.2 using these directions. ./bin/build_native_library.sh seems to succeed, and libaccumulo.so winds up in lib/native/libaccumulo.so in the Accumulo install directory....
Lindsey Kuper's user avatar
1 vote
1 answer
1k views

zfs for Hadoop cloud instead of ext4 [closed]

Right now I have couple of linodes with ext4. I have a hadoop setup. What benefit would I get if I migrate my file system from ext4 to zfs. Will there be any benefit in response times? Any speed ...
M-BoB's user avatar
  • 11
0 votes
1 answer
841 views

Server full ram but not show which process is use much RAM

One of our Hadoop datanode with CentOS7 64 bit is using all of it's RAM, I tried to figure out which process is using so much RAM but couldn't. Please help me to check this please: System: CentOS 7 ...
ZTE.A's user avatar
  • 1
5 votes
1 answer
2k views

Hadoop - Name Node and Data Node on the same machine

We have 7 identical physical servers (2x8 core CPU, 128GB RAM, 8x 6TB disks) that will be used for Hadoop. All of the machines are connected to 10G switch with double 10G interfaces. Since we do not ...
Merve Aydınlılar's user avatar
1 vote
0 answers
875 views

Zombie process blocking port when restarting Hadoop (Secondary) Namenode

I'm having weird issues with the Hadoop Namenode and Secondary Namenode. Our HDFS cluster runs smoothly most of the time. But every now and then, either the Primary Namenode freezes (crashing the ...
Janek Bevendorff's user avatar
4 votes
2 answers
10k views

Execute shell script as one of the steps on EMR AWS

We are thinking to migrate our Hadoop infrastructure from Data Center to AWS EMR. As some of the tasks / stages in ETL process are dependent e.g. flow is like Map Reduce job will generate data Shell ...
Free Coder's user avatar