Tag Info

Hot answers tagged

23

Straight from the horse's mouth: Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments ...


13

Use real server cases with good cooling/ventilation paths.


12

It would be incredibly bad. Hadoop eats heap like an elephant. The default heap size is 1000 MB but pretty much everybody increases it. 256 MB of RAM will not get you far, after the GPU and operating system take their toll, you'd probably have to restrict Java to 128 MB and the smallest jobs will run out of heap. The JVM is also very slow on ARM. Red Hat ...


10

Here's what I'd do. Get some large sheets of thin Lexan (or even foam board / corrugated plastic board - I know this as Correx, YMMV). Punch some holes through it, and bolt it to the holes on the uprights. Build about a foot or more higher on the top, cut out a hole and then mount a 12v automotive radiator fan on the top. You'll also want to build in ...


9

Thoughts: If you are asking these question on a public forum, hire experts to do it for you Consider Postgres and SQL Server which will scale to this volume too Do you need ACID? No = consider NoSQL Design and hardware matter more than the platform Don't virtualise or cut other hardware corners What is your RPO/RTO? Maintenance window? a.k.a are you really ...


9

Cloudera have some great videos that explain the principles behind Map Reduce and Hadoop. http://www.cloudera.com/hadoop-training-basic One of the core ideas behind MapReduce is that for large data sets you are going to be io bound on your disks, so in Hadoop HDFS gives you the ability to split things up between lots of nodes enabling parallel processing. ...


7

As the message tells you, any resource specification requires a name. In your specific case the following snippet should work: define hadoop_slave( $mem, $cpu, $dirs ) { hadoop_base { "${name}_hadoop_base": mem => $mem, cpu => $cpu, } hdfs_site { "${name}_hdfs_site": dirs => $dirs, } }


6

Hadoop is a scalable data processing platform. MySQL is a relational database for realtime usage. HBase (running on HDFS --> Hadoop Distributed File System) is a BigTable implementation: Big tables but not relational! Can you map your application into these constraints? Only if you can do this mapping then the question of "will it perform" is a useful ...


5

No. You can't suspend a job. There isn't. Slots are assigned for the duration of the tasks. You can decrease the priority in the scheduler which will allow some slots to be reclaimed as tasks complete.


5

If you can I would look at utilizing Cloud Infrastructure Services like Amazon Web Services (AWS) Elastic Compute Cloud (EC2), at least until you determine that it makes sense to invest in your own hardware. It's easy to get caught up in buying the shiny gear (I have to resist daily). By trying before you buy in the cloud you can learn a lot and answer the ...


5

The best thing you can do is get the disks replaced. The cost of disks won't weigh up against the cost of the cluster being down and your amount of work time being put in to fix the bad blocks. So even without a budget I would seriously try to convince your management.


3

For the ambari-server-setup exec you define a path containing /usr/bin and /usr/sbin. Most likely you have your bash binary installed as /bin/bash, hence you will also need to include /bin in that path.


3

It is required during the installation and upgrade procedure when using the wizard in the Cloudera Manager. During normal cluster operation it is not needed. All communication is done via the locally running agents then. In fact Cloudera Manager does not store the credentials, after the installation / upgrade it will delete it. When you upgrade your ...


3

If you have netcat installed. You could do echo stat | nc localhost 2181 | grep Mode echo srvr | nc localhost 2181 | grep Mode #(From 3.3.0 onwards) If the zookeper server is a leader, Returns Mode: leader Else returns Mode: follower


3

It should be safe enough to simply remove the Secondary Namenode role, and then add it back again onto a different node on the cluster. In the intermediate period, you may see a warning from Cloudera Manager that the role doesn't exist (which may cause long term issues with the namenode), but not having a Secondary doesn't put your data any more at risk.


3

Let me jsut say: cheaper to buy better boxes. No joke - 8 cores, 10g ram is a low end server. There ARE 2 providers of "combined VM technology" but they are commercial and it COSTS. Like it is needed of you neded t o combine high end boxes. Single core license costs more than your outdated desktops combined, sadly. So, the asnwer is: no way. Check MOSIX ...


3

Sounds like you've got a mountain of knowledge to climb, I doubt you'll be able to adaquately design what you're looking for without learning all the technologies for a solid year or so first. That said, I'd forget using something fancy like Hadoop. You didn't mention what kind of storage you have, but I'd try to pull some sort of SAN together, possibly ...


3

If you're using the default Derby database for storing the Hive metadata, you must run Hive as a user with write permissions to the database file. You can use any database with a JDBC driver for storing the metadata; if you're running a dev environment it's fine but anything that gets more usage should move to a different database. Choices include MySQL, ...


3

You can do this by setting -Ddfs.block.size=something with your hadoop fs command. For example: hadoop fs -Ddfs.block.size=1048576 -put ganglia-3.2.0-1.src.rpm /home/hcoyote As you can see here, the block size changes to what you define on the command line (in my case, the default is 64MB, but I'm changing it down to 1MB here). :; hadoop fsck -blocks ...


3

Simply create the file and add your content in it. See this. Use gksudo gedit /etc/apt/apt.conf for graphical edit, or sudoedit /etc/apt/apt.conf for console edit. Regarding what you have to put in the configuration, you have to put the proxy address and its port (so fill in the blanks). If you do need a proxy, ofc.


3

Looks like I had mistakenly set fs.default.name to 9000. Changed it to 8020, now everything works fine. Thanks to mgorven for the help!


3

Multiple JobTracker can be useful in Multi-Cluster architecture. So the cluster level load can be distributed between the JobTrackers. In a single cluster, following could become issues. (a) If multiple JobTracker servers will share an HDFS cluster, each must have a different mapred.system.dir, or the JobTrackers will delete each other's job files. (b) ...


3

For an automatic deploy you'll need to dive into the world of Chef and Crowbar. Probably the best place to start is here. http://robhirschfeld.com/2011/11/29/hadoop-crowbar-released-to-open-source For a manual deploy, the folks over at Big Data Craft have implemented Hadoop on OpenStack’s Swift. You'll need to use their implementation of java-cloudfiles. ...


3

/sbin/init is always the first process launched on your system. If the binary doesn't exists, the system cannot start. I don't know what happened, but it seems it has been erased from your disk, the easiest way to repair it is to reinstall the entire system. If you did correctly install your system first, you'll just have to reinstall the root partition, and ...


3

I would recommend to keep your nodes as similar as possible. As you have found out, different setups for each node make life difficult. The clusters I currently run each have every box exactly the same, which means all my configuration for every node is the same. Configuration is stored on an NFS homedir. The machines are installed as standard CentOS, and ...


3

The Smart Array 410/i controller only allows 2 raid devices to be created without the optional memory card added to it. So if your hardware only has 2 drives, then it would be fine; you can just make it 2 raid0's. If you have more then 2 drives you will probably have to purchase Part #462974-001 - 256MB Battery Backed Write Cache BBWC memory module. If you ...


3

We deployed hadoop. You can specify replication numbers for files. How many times a file gets replicated. Hadoop has a single point of failure on the namenode. If you are worried about disks going out, increase replication to 3 or more. Then if a disk goes bad, it's very simple. Throw it out and reformat. Hadoop will adjust automatically. In fact as ...


3

You mentioned this system was inherited (possibly not up to date) and that the load shoots up indicating a possible infinite loop. Does this bug report describe your situation? https://issues.apache.org/jira/browse/HDFS-466 If so, it's been reported as fixed in the latest HDFS 0.21.0 (just released last week): ...


2

MongoDB is simple and now has auto sharding. It is not very efficient for disk usage, so you will need to give it a lot of disk space. It can handle queries, but it will need indexes unless you want it to scan billions of records. What we have done is to actually store summaries in Mongo because if you know your query you can build an optimized data ...


2

Streaming just implies that it can offer you a constant bitrate above a certain threshhold when transferring the data, as opposed to having the data come in in bursts or waves. If HDFS is laid out for streaming, it will probably still support seek, with a bit of overhead it requires to cache the data for a constant stream. Of course, depending on system ...



Only top voted, non community-wiki answers of a minimum length are eligible