Reputation
Top tag
Next privilege 200 Rep.
See reduced ads
Badges
1 13
Impact
~14k people reached

  • 0 posts edited
  • 0 helpful flags
  • 126 votes cast
Jul
5
accepted How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?
Jul
5
answered How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?
Jul
5
comment How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?
Unfortunately, I can't recall what I was seeing in the datanode/namenode logs, which I'm pretty sure I would have checked. My "solution" though, which I I can't recall how I did (probably via dfsadmin), was to identify the files with missing blocks, and after concluding that we could live with their loss, I deleted the files from HDFS. In my case, there was a node/drive that died and replication was set to very low (possibly 1) on the troublesome files. I'll add this as an answer so as not to leave this question unanswered.
Nov
12
awarded  Notable Question
Nov
11
awarded  Popular Question
Sep
3
awarded  Popular Question
Jul
3
asked HBase issue with *some* regionservers connecting
Jul
2
awarded  Curious
Jun
9
awarded  Popular Question
Sep
19
comment Quickest way to transfer 55GB of images to new server
How can you store the compressed file(s) with rsync?
Aug
11
revised How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?
gave new update
Aug
10
comment How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?
Most files had replication=3 (the default), some larger datasets had replication=2, and it's possible some users used other replication (e.g. rep=1), but I'm not sure. With rep=2, HDFS should hopefully put that data on different servers, so I'm hoping that someone used rep=1 for some non-important data, and that's what the missing blocks are. I'll email the users to check
Aug
10
asked How to fix Hadoop HDFS cluster with missing blocks after one node was reinstalled?
Aug
5
awarded  Popular Question
Jun
22
awarded  Popular Question
Jan
25
awarded  Teacher
Jan
25
answered What are best practices for managing SSH keys in a team?
Jan
23
comment Bash Loop - How to stop the loop when I press Control-C inside a command?
Additional tip: if you're in a nested loop, and want to completely terminate, use break 2 (or replace "2" with the number of nested loops you want to terminate).
Jan
17
comment Performance impact of running different filesystems on a single Linux server
Maybe try emailing/tweeting the author.. let's get his/her input!
Jun
22
comment How much power supply do I need for my server, and could a shortage be causing my odd crashing?
In my case, the 5400's were not being used by Hadoop (more for long term less-accessed data). I'll look into sar and the IO wait time. Thanks