we have Hadoop version - 2.6.4

On the datanode machine we can see that hdfs data isn’t balanced

On some disks we have different used size as sdb 11G and sdd 17G

/dev/sdd 20G 3.0G 17G 15%   /grid/sdd 
/dev/sdb 20G 11G 9.3G 53%   /grid/sdb <-- WHY DISK DISK NOT BALANCED AS SDD DISK , WHY DISKS ARE DIFF USED SIZE!!!

After searching in google I found the following CLI ( from https://community.hortonworks.com/questions/19694/help-with-exception-from-hdfs-balancer.html )

hdfs balancer -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 20 1>/tmp/balancer-out.log 2>/tmp/balancer-debug.log

and after I run it we get the same hdfs size

/dev/sdd 20G 3.0G 17G 15% /grid/sdd 
/dev/sdb 20G 11G 9.3G 53% /grid/sdb

more /tmp/balancer-out.log Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved The cluster is balanced. Exiting... Mar 7, 2019 5:02:34 PM 0 0 B 0 B 0 B Mar 7, 2019 5:02:34 PM Balancing took 1.453 seconds

So actually we not get balanced in hdfs

Please advise , how to balance the hdfs data so all disk will be with the same used size

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.