6

I would like to know about your strategies on what to do when one of the Hadoop server disk fails.

Let's say, I have multiple (>15) Hadoop servers and 1 namenode, and one from 6 disks on slaves stops working, disks are connected via SAS. I don't care about retrieving data from this disk, but for general strategies for keeping cluster running.

What do you do?

5
  • Are you referring to a disk failure causing RAID degradation, or are you just not using redundant disks?
    – Kyle Smith
    Jun 26, 2010 at 21:24
  • @Kyle Smith: I'm not using redundant disks (no RAID of any kind)
    – wlk
    Jun 27, 2010 at 14:34
  • 1
    I threw in a bounty, same deal, hadoop, bunch of nodes, each node has several disks, mounted as /data/[a,b,c,d,e,f] I loose a disk, it means I loose the node until that is fixed. It sure looks like that shouldn't happen, but I'm not a hadoop expert, yet. Aug 31, 2010 at 22:57
  • @Ronald Pottol Did you notice my answer to your issue?
    – Rob Olmos
    Sep 7, 2010 at 20:25
  • Yes, I just wanted to check with my devs first. Sep 7, 2010 at 20:39

2 Answers 2

3

We deployed hadoop. You can specify replication numbers for files. How many times a file gets replicated. Hadoop has a single point of failure on the namenode. If you are worried about disks going out, increase replication to 3 or more.

Then if a disk goes bad, it's very simple. Throw it out and reformat. Hadoop will adjust automatically. In fact as soon as a disk goes out, it will start rebalancing files to maintain the replication numbers.

I am not sure why you have such a large bounty. You said you don't care to retrieve data. Hadoop only has a single point of failure on the name node. All other nodes are expendable.

5
  • Well, I am a different guy (one person asked, and a completely unrelated person put up the bounty months later, as they have the same issue). I know the data is replicated 3x, so nothing is lost, but losing a disk seems to be much worse than it should be for us. New team who inherited a poorly documented system. Our node tends to go nuts (load over 200) when they lose a disk, which seems unreasonable. I am hoping that a better config will fix that. Sep 1, 2010 at 6:42
  • Are you interested in keeping a node alive after a disk goes bad? I agree the node should gracefully die instead of going nuts with a load over 200. So is this only a Hadoop bug of why doesn't a node die when a disk goes out? Hadoop's architecture is simple, if a node dies, take it out. Add another one later if you like or put the same one back in when it is fixed.
    – Amala
    Sep 1, 2010 at 13:38
  • Yes, after all, they have something like 6+ disks for hadoop, each on it's own file system, so loosing a disk just means some data needs to be replicated from the other two copies, but no reason for the node and the other 5 disks of data to go off line. Sep 1, 2010 at 17:30
  • A datanode shouldn't die with the loss of a single drive. I suspect that somebody did something silly on your machine like striped the disks. Hadoop is much better off if you mount all drives independently with no RAID whatsoever. In this situation, losing a drive causes re-replication, but doesn't take out the datanode. Dec 5, 2010 at 20:22
  • No, no raid (well, the OS is on raid, but not the hadoop data). Dec 22, 2010 at 0:40
3
+150

You mentioned this system was inherited (possibly not up to date) and that the load shoots up indicating a possible infinite loop. Does this bug report describe your situation?

https://issues.apache.org/jira/browse/HDFS-466

If so, it's been reported as fixed in the latest HDFS 0.21.0 (just released last week):

http://hadoop.apache.org/hdfs/docs/current/releasenotes.html

Disclaimer: To my disappointment I have yet to have the need to use Hadoop/HDFS :)

1
  • That bug refers to C libraries, not the normal Java libraries, I think. Dec 5, 2010 at 20:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .