Tag Info

Hot answers tagged

23

Straight from the horse's mouth: Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments ...


13

Use real server cases with good cooling/ventilation paths.


12

It would be incredibly bad. Hadoop eats heap like an elephant. The default heap size is 1000 MB but pretty much everybody increases it. 256 MB of RAM will not get you far, after the GPU and operating system take their toll, you'd probably have to restrict Java to 128 MB and the smallest jobs will run out of heap. The JVM is also very slow on ARM. Red Hat ...


10

Here's what I'd do. Get some large sheets of thin Lexan (or even foam board / corrugated plastic board - I know this as Correx, YMMV). Punch some holes through it, and bolt it to the holes on the uprights. Build about a foot or more higher on the top, cut out a hole and then mount a 12v automotive radiator fan on the top. You'll also want to build in ...


9

Thoughts: If you are asking these question on a public forum, hire experts to do it for you Consider Postgres and SQL Server which will scale to this volume too Do you need ACID? No = consider NoSQL Design and hardware matter more than the platform Don't virtualise or cut other hardware corners What is your RPO/RTO? Maintenance window? a.k.a are you really ...


8

Cloudera have some great videos that explain the principles behind Map Reduce and Hadoop. http://www.cloudera.com/hadoop-training-basic One of the core ideas behind MapReduce is that for large data sets you are going to be io bound on your disks, so in Hadoop HDFS gives you the ability to split things up between lots of nodes enabling parallel processing. ...


7

As the message tells you, any resource specification requires a name. In your specific case the following snippet should work: define hadoop_slave( $mem, $cpu, $dirs ) { hadoop_base { "${name}_hadoop_base": mem => $mem, cpu => $cpu, } hdfs_site { "${name}_hdfs_site": dirs => $dirs, } }


6

Hadoop is a scalable data processing platform. MySQL is a relational database for realtime usage. HBase (running on HDFS --> Hadoop Distributed File System) is a BigTable implementation: Big tables but not relational! Can you map your application into these constraints? Only if you can do this mapping then the question of "will it perform" is a useful ...


5

The best thing you can do is get the disks replaced. The cost of disks won't weigh up against the cost of the cluster being down and your amount of work time being put in to fix the bad blocks. So even without a budget I would seriously try to convince your management.


5

If you can I would look at utilizing Cloud Infrastructure Services like Amazon Web Services (AWS) Elastic Compute Cloud (EC2), at least until you determine that it makes sense to invest in your own hardware. It's easy to get caught up in buying the shiny gear (I have to resist daily). By trying before you buy in the cloud you can learn a lot and answer the ...


4

No. You can't suspend a job. There isn't. Slots are assigned for the duration of the tasks. You can decrease the priority in the scheduler which will allow some slots to be reclaimed as tasks complete.


3

I would recommend to keep your nodes as similar as possible. As you have found out, different setups for each node make life difficult. The clusters I currently run each have every box exactly the same, which means all my configuration for every node is the same. Configuration is stored on an NFS homedir. The machines are installed as standard CentOS, and ...


3

Let me jsut say: cheaper to buy better boxes. No joke - 8 cores, 10g ram is a low end server. There ARE 2 providers of "combined VM technology" but they are commercial and it COSTS. Like it is needed of you neded t o combine high end boxes. Single core license costs more than your outdated desktops combined, sadly. So, the asnwer is: no way. Check MOSIX ...


3

Sounds like you've got a mountain of knowledge to climb, I doubt you'll be able to adaquately design what you're looking for without learning all the technologies for a solid year or so first. That said, I'd forget using something fancy like Hadoop. You didn't mention what kind of storage you have, but I'd try to pull some sort of SAN together, possibly ...


3

We deployed hadoop. You can specify replication numbers for files. How many times a file gets replicated. Hadoop has a single point of failure on the namenode. If you are worried about disks going out, increase replication to 3 or more. Then if a disk goes bad, it's very simple. Throw it out and reformat. Hadoop will adjust automatically. In fact as ...


3

For an automatic deploy you'll need to dive into the world of Chef and Crowbar. Probably the best place to start is here. http://robhirschfeld.com/2011/11/29/hadoop-crowbar-released-to-open-source For a manual deploy, the folks over at Big Data Craft have implemented Hadoop on OpenStack’s Swift. You'll need to use their implementation of java-cloudfiles. ...


3

/sbin/init is always the first process launched on your system. If the binary doesn't exists, the system cannot start. I don't know what happened, but it seems it has been erased from your disk, the easiest way to repair it is to reinstall the entire system. If you did correctly install your system first, you'll just have to reinstall the root partition, and ...


3

The Smart Array 410/i controller only allows 2 raid devices to be created without the optional memory card added to it. So if your hardware only has 2 drives, then it would be fine; you can just make it 2 raid0's. If you have more then 2 drives you will probably have to purchase Part #462974-001 - 256MB Battery Backed Write Cache BBWC memory module. If you ...


3

For the ambari-server-setup exec you define a path containing /usr/bin and /usr/sbin. Most likely you have your bash binary installed as /bin/bash, hence you will also need to include /bin in that path.


3

It is required during the installation and upgrade procedure when using the wizard in the Cloudera Manager. During normal cluster operation it is not needed. All communication is done via the locally running agents then. In fact Cloudera Manager does not store the credentials, after the installation / upgrade it will delete it. When you upgrade your ...


3

Simply create the file and add your content in it. See this. Use gksudo gedit /etc/apt/apt.conf for graphical edit, or sudoedit /etc/apt/apt.conf for console edit. Regarding what you have to put in the configuration, you have to put the proxy address and its port (so fill in the blanks). If you do need a proxy, ofc.


2

Honestly? Unless its in a very dirty or very hot place (ambient temperature of above 30c in the room), nothing. Those computers aren't going to generate much heat and there is loads of room around them to dissipate the heat they do generate. Not nearly enough to worry about in any case.


2

You stated that dd is to slow for you. How are you using it? Because dding /dev/zero into some file seems fast to me, also you can reduce writes to disk (and thus effectively speed up) the dd with its obs parameter. You probably should first find, where the bottleneck of generating/writing data is.


2

Put following config in conf/hdfs-site.xml: <property> <name>dfs.hosts</name> <value>/full/path/to/whitelisted/node/file</value> </property> Use following command to ask Hadoop to refresh node status to based on configuration. ./bin/hadoop dfsadmin -refreshNodes


2

You can do this by setting -Ddfs.block.size=something with your hadoop fs command. For example: hadoop fs -Ddfs.block.size=1048576 -put ganglia-3.2.0-1.src.rpm /home/hcoyote As you can see here, the block size changes to what you define on the command line (in my case, the default is 64MB, but I'm changing it down to 1MB here). :; hadoop fsck -blocks ...


2

Check the hadoop-env.sh file in the Hadoop conf directory. Ensure the line exporting JAVA_HOME has been uncommented and set appropriately.


2

/usr/share is a platform-independent shared data directory (and hadoop is in java...) and /etc/hadoop is mostly host specific system and application configuration files. Then, to configure hadoop, just use the following command and answer yes, yes, yes... ! sudo ./usr/sbin/hadoop-setup-single-node.sh Then sudo ./usr/sbin/hadoop-validate-setup.sh ...


2

How many horse powers does my car need? I recently added 50 kg weight. You see the problem? We do not know. YOU tell us. For example you talk so much about hard discs and Ubuntu, but do not say memory (uses power) and PROCESSOR. 430 watt is WAY too little for a high end processor - but may work for an atom. It may even work for a single processor, but not ...


2

Looks like I had mistakenly set fs.default.name to 9000. Changed it to 8020, now everything works fine. Thanks to mgorven for the help!


2

Multiple JobTracker can be useful in Multi-Cluster architecture. So the cluster level load can be distributed between the JobTrackers. In a single cluster, following could become issues. (a) If multiple JobTracker servers will share an HDFS cluster, each must have a different mapred.system.dir, or the JobTrackers will delete each other's job files. (b) ...



Only top voted, non community-wiki answers of a minimum length are eligible