I would like to know about your strategies on what to do when one of the Hadoop server disk fails.

Let's say, I have multiple (>15) Hadoop servers and 1 namenode, and one from 6 disks on slaves stops working, disks are connected via SAS. I don't care about retrieving data from this disk, but for general strategies for keeping cluster running.

What do you do?

  • Are you referring to a disk failure causing RAID degradation, or are you just not using redundant disks? – Kyle Smith Jun 26 '10 at 21:24
  • @Kyle Smith: I'm not using redundant disks (no RAID of any kind) – wlk Jun 27 '10 at 14:34
  • 1
    I threw in a bounty, same deal, hadoop, bunch of nodes, each node has several disks, mounted as /data/[a,b,c,d,e,f] I loose a disk, it means I loose the node until that is fixed. It sure looks like that shouldn't happen, but I'm not a hadoop expert, yet. – Ronald Pottol Aug 31 '10 at 22:57
  • @Ronald Pottol Did you notice my answer to your issue? – Rob Olmos Sep 7 '10 at 20:25
  • Yes, I just wanted to check with my devs first. – Ronald Pottol Sep 7 '10 at 20:39

We deployed hadoop. You can specify replication numbers for files. How many times a file gets replicated. Hadoop has a single point of failure on the namenode. If you are worried about disks going out, increase replication to 3 or more.

Then if a disk goes bad, it's very simple. Throw it out and reformat. Hadoop will adjust automatically. In fact as soon as a disk goes out, it will start rebalancing files to maintain the replication numbers.

I am not sure why you have such a large bounty. You said you don't care to retrieve data. Hadoop only has a single point of failure on the name node. All other nodes are expendable.

  • Well, I am a different guy (one person asked, and a completely unrelated person put up the bounty months later, as they have the same issue). I know the data is replicated 3x, so nothing is lost, but losing a disk seems to be much worse than it should be for us. New team who inherited a poorly documented system. Our node tends to go nuts (load over 200) when they lose a disk, which seems unreasonable. I am hoping that a better config will fix that. – Ronald Pottol Sep 1 '10 at 6:42
  • Are you interested in keeping a node alive after a disk goes bad? I agree the node should gracefully die instead of going nuts with a load over 200. So is this only a Hadoop bug of why doesn't a node die when a disk goes out? Hadoop's architecture is simple, if a node dies, take it out. Add another one later if you like or put the same one back in when it is fixed. – Amala Sep 1 '10 at 13:38
  • Yes, after all, they have something like 6+ disks for hadoop, each on it's own file system, so loosing a disk just means some data needs to be replicated from the other two copies, but no reason for the node and the other 5 disks of data to go off line. – Ronald Pottol Sep 1 '10 at 17:30
  • A datanode shouldn't die with the loss of a single drive. I suspect that somebody did something silly on your machine like striped the disks. Hadoop is much better off if you mount all drives independently with no RAID whatsoever. In this situation, losing a drive causes re-replication, but doesn't take out the datanode. – Ted Dunning Dec 5 '10 at 20:22
  • No, no raid (well, the OS is on raid, but not the hadoop data). – Ronald Pottol Dec 22 '10 at 0:40

You mentioned this system was inherited (possibly not up to date) and that the load shoots up indicating a possible infinite loop. Does this bug report describe your situation?


If so, it's been reported as fixed in the latest HDFS 0.21.0 (just released last week):


Disclaimer: To my disappointment I have yet to have the need to use Hadoop/HDFS :)

  • That bug refers to C libraries, not the normal Java libraries, I think. – Ted Dunning Dec 5 '10 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.