Our HDFS cluster is only 90% full but some datanodes have some disks that are 100% full. That means when we mass reboot the entire cluster some datanodes completely fail to start with a message like this:

2013-10-26 03:58:27,295 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Mkdirs failed to create /mnt/local/sda1/hadoop/dfsdata/blocksBeingWritten

Only three have to fail this way before we start experiencing real data loss.

Currently we workaround it by decreasing the amount of space reserved for the root user but we'll eventually run out. We also run the re-balancer pretty much constantly, but some disks stay stuck at 100% anyway.

Changing the dfs.datanode.failed.volumes.tolerated setting is not the solution as the volume has not failed.

Any ideas?

  • 1
    Increase the size of the disks/volumes? – joeqwerty Oct 29 '13 at 15:09

As per this default HDFS parameter, the dfs.datanode.du.reserved is per volume. So if you set it to say 10 GB and your datanode is having 4 volumes configured for HDFS, it will set aside 40 GB for non DFS use.


Perhaps you should move data from one volume to another within one datanode manually:

3.12. On an individual data node, how do you balance the blocks on the disk?

Hadoop currently does not have a method by which to do this automatically. To do this manually:

  1. Shutdown the DataNode involved
  2. Use the UNIX mv command to move the individual block replica and meta pairs from one directory to another on the selected host. On releases which have HDFS-6482 (Apache Hadoop 2.6.0+) you also need to ensure the subdir-named directory structure remains exactly the same when moving the blocks across the disks. For example, if the block replica and its meta pair were under /data/1/dfs/dn/current/BP-1788246909-, and you wanted to move it to /data/5/ disk, then it MUST be moved into the same subdirectory structure underneath that, i.e. /data/5/dfs/dn/current/BP-1788246909- If this is not maintained, the DN will no longer be able to locate the replicas after the move.
  3. Restart the DataNode.

Source: https://wiki.apache.org/hadoop/FAQ


for your case , balancing your data evenly over the cluster datanodes might help you to avoid disks getting full even if overall cluster has space you can force run hadoop oob balancer periodically.this will shuffle blocks so all datanodes will be consuming same amount of disk space

    hadoop balancer

you can specify threshold to this script to which hdfs blocks will be balanced , default is 10 , you can specify lower number if you think 10% is not enough , I see you already using 90% space on hdfs ,so you can go for 5

    hadoop balancer 5 

check these link1 link2 also for more on balancer

  • Did you see this part of my question? "We also run the re-balancer pretty much constantly, but some disks stay stuck at 100% anyway." – mbac Nov 19 '13 at 15:00
  • already tried minimizing threshold ? to 5 or below – Ratnakar Pawar Dec 6 '13 at 13:51

I think what you really want is to set dfs.datanode.du.reserved to some non-zero value, so that the datanode ensures there will always be that much space free on the system's HDFS volumes.

Note: dfs.datanode.du.reserved is for freespace on the entire system, not per volume.

According to HDFS-1564, there's discussion on making a per-volume option, but it's not slated for any particular version yet.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.