0

We recently added new nodes to the cluster and I want to run the re-balancer to distribute the data among these nodes. We are using CDH3 and whole system is cheffed - I am not even sure if there is cloudera manager installed on any of these boxes.

Will I be able to run the balancer if without Cloudera Manager? What is the right way of doing the redistribution?

Please pardon if this is really obvious question, I googled a lot and searched on questions here as well without success, hence these questions.

PS: I had no option but to put this in CDH4 as it won't let me create CDH3 tag.

0

Yes, you can run the balancer without CM. You will need to do something like the following, preferably running from your namenode, but it should work from any node or client that has access.

First, run screen or tmux. This process can take awhile. Nothing bad happens if you DON'T run this in screen/tmux, this is just protection for if your connection to the remote system drops.

If you're not running Kerberos, you can do this:

sudo su - hdfs
hadoop balancer -threshold <somevalue> > balance.out 2>&1

If you ARE running Kerberos, you will need to do something like this:

sudo su - hdfs
kinit -t -k /path/to/your/hdfs.keytab hdfs/fully.qualified.hostname
hadoop balancer -threshold <somevalue> > balance.out 2>&1

and then in a different shell, you can tail the output, passing it through appropriate greps to weed out the useless information, which the balancer produces A LOT of.

I use something like this:

tail -f balance.out | grep -v Moving

which means I will see information like this:

13/11/25 05:53:38 INFO balancer.Balancer: 0 over utilized nodes:
13/11/25 05:53:38 INFO balancer.Balancer: 1 under utilized nodes:  192.168.1.151:50010
13/11/25 05:53:38 INFO balancer.Balancer: Need to move 181.53 MB bytes to make the cluster balanced.
13/11/25 05:53:38 INFO balancer.Balancer: Decided to move 10 GB bytes from 192.168.1.131:50010 to 192.168.1.151:50010
13/11/25 05:53:38 INFO balancer.Balancer: Will move 10 GBbytes in this iteration
Nov 25, 2013 5:53:38 AM        2203             47.76 TB           181.53 MB              10 GB
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.147:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.122:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.137:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.128:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.145:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.126:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.149:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.146:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.153:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.156:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.151:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.134:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.135:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.154:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.144:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.125:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.148:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.139:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.152:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.133:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.132:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.136:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.150:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.129:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.130:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.142:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.123:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.127:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.160:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.158:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.131:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.138:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.124:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-bottom/192.168.1.140:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.159:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a8-top/192.168.1.121:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.155:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.141:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.157:50010
13/11/25 05:54:23 INFO net.NetworkTopology: Adding a new node: /c1/hadoop-a6/192.168.1.143:50010
13/11/25 05:54:23 INFO balancer.Balancer: 0 over utilized nodes:
13/11/25 05:54:23 INFO balancer.Balancer: 0 under utilized nodes: 
The cluster is balanced. Exiting...
Balancing took 90.16988833333333 hours

Finally, if you feel the rebalance is taking too long, you can tweak dfs.balance.bandwidthPerSec. It needs to be set on each datanode and the hadoop-0.20-datanode process restarted to take affect. The default is 1MB/s as I recall. The value is specified in bytes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.