Questions tagged [hdfs]
For questions regarding the Hadoop distributed file system (HDFS) which is part of the Apache Hadoop project.
74
questions
1
vote
0
answers
44
views
Hadoop + warnings as slow block-receive from data-node machines
We have Hadoop cluster with 487 data-nodes machines ( each data-node machine include also the Service node-manager ) , all machines are physical machines ( DELL ) , and OS is RHEL 7.9 version.
Each ...
- 617
0
votes
0
answers
39
views
Hadoop Namenode heap size tuning
NameNode process is executed in Java virtual machine,
and Java object which NameNode creates is managed in Java virtual memory. As the fles or directories are created, inode objects and block objects ...
- 617
0
votes
0
answers
85
views
HDFS + Taking long time to copy files from hdfs to local folder
we have Hadoop cluster that used HDFS filesystem
the problem is about copying file from HDFS to local folder that take huge time
for example usually copy file with size of 3G takes few second , but ...
- 617
0
votes
1
answer
103
views
Does VM machine can replace physical machine,
We have 254 Physical servers when all machines are DELL servers R740.
servers are part of Hadoop cluster. most of them are holding HDFS filesystem and data node & node manager services, part of ...
- 617
1
vote
0
answers
175
views
HDP cluster + journal nodes get out of Sync
we have HDP cluster version 2.6.5
when we look on name-node logs we can see the following warning
2023-02-20 15:56:37,731 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(...
- 617
-2
votes
1
answer
134
views
How does placing data in various racks help to exploit the fact that intra-rack aggregated bandwidth>=inter-rack bandwidth?
GFS research paper snapshot
it says that(my interpretation after reading research paper and its reviews) "inter rack bandwidth is lower than aggregated intra rack bandwidth(not sure what it means ...
1
vote
0
answers
135
views
HDFS + how to disable the "du -sk" verifcation on data node disks
We are using HDP cluster with 182 data node machines:
HDP version - 2.6.4
Ambari version 2.6.1
We note the following behavior on the data nodes machines (its happens on all data-node machines and on ...
- 617
0
votes
1
answer
116
views
Hadoop recommissioning datanode
Do I need to delete all data from a datanode before recommissioning it, or it doesn't matter and the namenode will not pick stale data from the datanode?
0
votes
1
answer
293
views
Change HDFS replication factor
I've changed replication factor from 3 to 2 for some directories with command:
hdfs dfs -setrep -R 2 /path/to/dir
but my HDFS free space still the same. Should I do something else to free my disks?
0
votes
1
answer
93
views
HDFS. How to free 1 particular disk
I have cluster with 3 servers. 2 of them have 2 TB disks and another one have 500 Gb SSD. I am trying to use balancer, but I still get 70% of usage on 2TB disks and 99% on 500Gb due to non-dfs files. ...
0
votes
1
answer
224
views
Hadoop Cluster Capacity Planning of Data Nodes for disks per data node
we are planing to build hadoop cluster with 12 data nodes machines
when the replication factor is 3
and DataNode failed disk tolerance - 1
data nodes machines are include the disks for HDFS
since we ...
- 617
0
votes
1
answer
264
views
Optimal RAID configuration for EC2 instance store used for HDFS
I'm trying to determine if there is any practical advantage to configuring a RAID array on the instance store of a 3x d2.2xlarge instances being used for HDFS. Initially I planned to just mount each ...
- 393
1
vote
1
answer
1k
views
List all files in hdfs directory
Due to some error at one component, files in HDFS got accumulated and the number is huge i.e 2123516. I want to list all files and want to copy their name in one file but when I run the following ...
- 21
0
votes
1
answer
1k
views
AWS FSx for lustre with S3 vs EMR (with EMRFS) for spark jobs
We are currently using EMR for easy job submission for our spark jobs.
Recently I came across the "FSx lustre + S3" solution that is being advertised as ideal for HPC situations.
EMRFS however is also ...
- 245
0
votes
1
answer
229
views
is it possible mix different RHEL OS version in hadoop cluster?
we are using the following HDP cluster with ambari ,
list of nodes and their RHEL version
3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2
312 DATA-NODES machines ...
- 501
0
votes
1
answer
2k
views
HDFS block deletion speed - cause, expectance, tuning?
I have a small (testing) HDFS cluster which I use as snapshot backup space for Flink. Flink creates and deletes roughly 1000 (small) files per second. The namenode seems to handle this without ...
- 111
0
votes
0
answers
220
views
Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?
I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
- 101
1
vote
0
answers
369
views
HDFS NFS gateway read Input/output error
I have enabled the HDFS NFSv3 gateway on our HDFS cluster through official documentation. Everything works well except for one Ubuntu 16.04 server machine. The following is the kernel, mount and ...
- 21
0
votes
1
answer
730
views
HDFS balancing , how to balanced hdfs data?
we have Hadoop version - 2.6.4
On the datanode machine we can see that hdfs data isn’t balanced
On some disks we have different used size as sdb 11G and sdd 17G
/dev/sdd 20G 3.0G 17G 15% /grid/...
- 501
1
vote
1
answer
292
views
Hadoop: Failed to start backup node, bad state: DROP_UNTIL_NEXT_ROLL
I have created a small Hadoop cluster setup with 1 NameNode and 1 DataNode to get hands-on.
below is my configuration files:
Core-site.xml
<property>
<name>fs.defaultFS</name>
...
- 111
1
vote
0
answers
101
views
Sample output of Rumen or Input to Gridmix
I want to see JobHistory logs, which can be fed as input to the Rumen. More specifically, I am interested in knowing input format for the Gridmix.
I tried following two things for it:
1) I found ...
- 111
1
vote
0
answers
192
views
ambari cluster + when need to set Block replication to 1
We get the following in Spark logs:
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
- 501
0
votes
0
answers
42
views
Datanode machines disks size
is it important that ( workers ) datanode machines disks will be with the same size?
for example
we have ambari cluster with 3 workers machines ( datanode machines )
each datanode machine have 10 ...
- 501
0
votes
1
answer
454
views
what is effected when running - hadoop namenode -format
we have amabri cluster ( version 2.6 ) with 24 workers machines
we want to run following commands only on worker23 machine ( because problem on worker23 ) , dose these commands effected on all ...
- 69
1
vote
0
answers
56
views
How to prevent arbitrary executable execution on hadoop cluster
I am involved with configuring a Hadoop cluster for complete auditability and security. I am new to the Hadoop ecosystem, but I have a decent idea of the basics. I have a few concerns for which I hope ...
- 111
4
votes
1
answer
23k
views
mount.nfs: mount system call failed
I am trying to mount hdfs on my local machine running Ubuntu using the following command :---
sudo mount -t nfs -o vers=3,proto=tcp,nolock 192.168.170.52:/ /mnt/hdfs_mount/
But I am getting this ...
- 141
0
votes
1
answer
1k
views
copying files in hdfs stalls
Have a 35 node cluster with a high number of blocks in it: ≈450K blocks per data node.
After configuration change (which contained rack reassignments and NameNode Xmx increase) HDFS became a problem. ...
- 101
0
votes
1
answer
148
views
how to install hadoop2.4.1 in windows with spark 2.0.0
i want to setup a cluster using hadoop in yarn mode..i want to use spark API for map-reduce and will use spark submit to deploy my applications..i want to work on cluster..can anyone help me how to ...
3
votes
0
answers
167
views
How can I launch hdfs on Mesos without DC/OS?
From my understand DC/OS is a freemium managed service. Because I'd rather just have a raw Mesos implementation, I'd rather not be dependent on DC/OS and so I just want to know how to implement HDFS ...
- 219
1
vote
0
answers
20
views
Need to set 000 permission to specific hdfs data block through commandline
I am trying to set the “000” permission to the specific block.
I used below command to find the block information:
su - hdfs -c "hdfs fsck -locations -files -blocks /user/rohit/...
2
votes
1
answer
2k
views
Running HDFS with only 1 data node - appending fails
I'm trying to test some services that require HDFS using Docker Compose. Since the services being tested, namenode, and data node(s) will all be running on the same physical machine (dev laptop), it ...
- 141
2
votes
2
answers
1k
views
Can HDFS be installed on ZFS or BTRFS hard disks?
I am new to hadoop and confused about how HDFS works with ZFS or BTRFS.
Can physical drives be mounted using ZFS and then have HDFS installed on top of ZFS?
Or can HDFS be installed directly?
- 1,677
1
vote
0
answers
27
views
Is it possible to configure hdfs in a federation mode and in an HA mode in the same time?
I don't understand if it possible to configure HDFS in both modes in the same time. Does it make sense? Can somebody show a simple configuration of HDFS in both modes? (nameNode1, nameNode2, ...
- 763
0
votes
1
answer
833
views
Why does DFSZKFailoverController kills Namenode process in hadoop?
I try to configure hadoop high availability cluster by following this tutorial:
http://www.edureka.co/blog/how-to-set-up-hadoop-cluster-with-hdfs-high-availability/
When I follow that article I faces ...
- 763
1
vote
0
answers
879
views
Zombie process blocking port when restarting Hadoop (Secondary) Namenode
I'm having weird issues with the Hadoop Namenode and Secondary Namenode.
Our HDFS cluster runs smoothly most of the time. But every now and then, either the Primary Namenode freezes (crashing the ...
- 186
1
vote
0
answers
2k
views
HDFS performances on apache spark
I have several issues related to HDFS, that may have different roots. I'm posting as much information as I can, with the hope that I can get your opinion on at least some of them. Basically the cases ...
- 133
4
votes
2
answers
7k
views
Upload large files with curl without RAM cache.
I'm using curl to upload large files (from 5 to 20Gb) to HOOP based on HDFS (Hadoop Cluster) as follows:
curl -f --data-binary "@$file" "$HOOP_HOST$UPLOAD_PATH?user.name=$HOOP_USER&op=create"
...
- 81
0
votes
1
answer
3k
views
Flume- Error Log while using FileChannel
I am using Flume flume-ng-1.5.0 ( with CDH 5.4) to collect logs from many Servers and Sink to HDFS
Here is my configuration :
#Define Source , Sinks, Channel
collector.sources = avro
collector....
- 214
1
vote
1
answer
81
views
Viewing NxN network traffic in a distributed system
I have a Hadoop cluster set up, and I would like to view the network usage between all nodes in this cluster, i.e. if I have N nodes, I want to see NxN network connections so that I can view all ...
- 233
4
votes
3
answers
21k
views
Is there a way to grep gzipped content in hdfs without extracting it?
I'm looking for a way to zgrep hdfs files
something like:
hadoop fs -zcat hdfs://myfile.gz | grep "hi"
or
hadoop fs -cat hdfs://myfile.gz | zgrep "hi"
it does not really work for me is there ...
- 711
0
votes
1
answer
2k
views
Hadoop: How to configure failover time for a datanode
I need to re-replicate blocks on my HDFS cluster in case of a datanode is failing. Actually, this appears to already happen after a period of maybe 10min. However, I want to decrease this time, but ...
- 573
7
votes
2
answers
2k
views
HBASE Space Used Started Climbing Rapidly
Update 4,215:
After looking at space usage inside of hdfs, I see that .oldlogs is using a lot of space:
1485820612766 /hbase/.oldlogs
So new questions:
What is it?
How do I clean it up?
How do ...
- 84.4k
1
vote
0
answers
562
views
How to force HDFS to use LDAP user's UID
I have a cloudera cluster with HDFS and Hue services and I'm trying to unify the authentication using LDAP.
I have my LDAP server running thanks to 389-ds (not sure if is the best way) and I can log ...
- 119
3
votes
1
answer
908
views
Linux Network tuning to prevent tcp rcvpruned and backlogdrop?
My datanodes in my hbase cluster are triggering some tcp rcvpruned and backlog drops from time to time:
It seems to be there are at least two angles to approach this at:
Tune HBase/HDFS etc... so ...
- 84.4k
1
vote
1
answer
2k
views
Rhadoop hdfs.init() Error
I recently installed CDH5.1.0 along with R 3.1.*, and I got rmr2, rJava, and rhdfs all installed properly. (along with the required packages and set the required environment variables) After some ...
- 11
2
votes
0
answers
4k
views
"The volume for a file has been externally altered ..." and setting file sizes on write to NFS from Windows
I have set up NFS access to HDFS.
I can connect to the default root share from both Windows and Linux clients and all clients can read from the share. Writing from the Linux client is also ok.
When ...
- 121
3
votes
2
answers
4k
views
Disable The Under Replicated Blocks Alert in Cloudera Manager
I have a single server Hbase cluster that I am only using as the sink end of HBase replication. Therefore I don't want to replicate any blocks within this cluster (since the source has replicated ...
- 84.4k
0
votes
2
answers
12k
views
error while running any hadoop hdfs file system command
I am very new to hadoop and referring to the "hadoop for dummies" book.
I have a VM with following specs: hadoop version 2.0.6-alpha bigtop os centos
The problem is when I run any hdfs file system ...
2
votes
2
answers
5k
views
Unable to convert HDFS from non-HA to HA
Introduction
Aim: Convert HDFS from non-HA to HA.
Method: According to this documentation it should be possible to convert HFDS from non- to HA by implementing following configuration:
/etc/hadoop/...
- 6,011
4
votes
0
answers
2k
views
java.lang.NullPointerException When Doing A Read in HDFS
I have had a 10 node HBase cluster up and running for the past 4 months. The cluster was setup on VMs in a corporate environment which I do not control, but everything has been working great...until ...
- 233