Tag Info

Hot answers tagged

23

Straight from the horse's mouth: Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments ...


13

Use real server cases with good cooling/ventilation paths.


12

It would be incredibly bad. Hadoop eats heap like an elephant. The default heap size is 1000 MB but pretty much everybody increases it. 256 MB of RAM will not get you far, after the GPU and operating system take their toll, you'd probably have to restrict Java to 128 MB and the smallest jobs will run out of heap. The JVM is also very slow on ARM. Red Hat ...


10

Here's what I'd do. Get some large sheets of thin Lexan (or even foam board / corrugated plastic board - I know this as Correx, YMMV). Punch some holes through it, and bolt it to the holes on the uprights. Build about a foot or more higher on the top, cut out a hole and then mount a 12v automotive radiator fan on the top. You'll also want to build in ...


9

Thoughts: If you are asking these question on a public forum, hire experts to do it for you Consider Postgres and SQL Server which will scale to this volume too Do you need ACID? No = consider NoSQL Design and hardware matter more than the platform Don't virtualise or cut other hardware corners What is your RPO/RTO? Maintenance window? a.k.a are you really ...


9

Cloudera have some great videos that explain the principles behind Map Reduce and Hadoop. http://www.cloudera.com/hadoop-training-basic One of the core ideas behind MapReduce is that for large data sets you are going to be io bound on your disks, so in Hadoop HDFS gives you the ability to split things up between lots of nodes enabling parallel processing. ...


8

It is possible to check whether a zookeeper server is a leader or follower using the nc command that is included in the netcat package: echo stat | nc localhost 2181 | grep Mode echo srvr | nc localhost 2181 | grep Mode #(From 3.3.0 onwards) If the zookeeper server is a leader the command will return: Mode: leader and otherwise: Mode: follower


7

My suggestion is to read the information provided. java.net.ConnectException: Connection refused; This generally means nothing is listening on the specified IP:Port. Note also this information is provided http://wiki.apache.org/hadoop/ConnectionRefused and it's first line You get a ConnectionRefused Exception when there is a machine at the ...


7

As the message tells you, any resource specification requires a name. In your specific case the following snippet should work: define hadoop_slave( $mem, $cpu, $dirs ) { hadoop_base { "${name}_hadoop_base": mem => $mem, cpu => $cpu, } hdfs_site { "${name}_hdfs_site": dirs => $dirs, } }


6

Hadoop is a scalable data processing platform. MySQL is a relational database for realtime usage. HBase (running on HDFS --> Hadoop Distributed File System) is a BigTable implementation: Big tables but not relational! Can you map your application into these constraints? Only if you can do this mapping then the question of "will it perform" is a useful ...


6

The best thing you can do is get the disks replaced. The cost of disks won't weigh up against the cost of the cluster being down and your amount of work time being put in to fix the bad blocks. So even without a budget I would seriously try to convince your management.


5

No. You can't suspend a job. There isn't. Slots are assigned for the duration of the tasks. You can decrease the priority in the scheduler which will allow some slots to be reclaimed as tasks complete.


5

If you can I would look at utilizing Cloud Infrastructure Services like Amazon Web Services (AWS) Elastic Compute Cloud (EC2), at least until you determine that it makes sense to invest in your own hardware. It's easy to get caught up in buying the shiny gear (I have to resist daily). By trying before you buy in the cloud you can learn a lot and answer the ...


4

Simply create the file and add your content in it. See this. Use gksudo gedit /etc/apt/apt.conf for graphical edit, or sudoedit /etc/apt/apt.conf for console edit. Regarding what you have to put in the configuration, you have to put the proxy address and its port (so fill in the blanks). If you do need a proxy, ofc.


4

CopyFromLocal does not have the ability to display the file copy progress. Alternatively, you could open another shell and run the $ watch hadoop fs -ls <filenameyouarecopying>. This will display the file and its size once every 2.0 seconds.


4

It is required during the installation and upgrade procedure when using the wizard in the Cloudera Manager. During normal cluster operation it is not needed. All communication is done via the locally running agents then. In fact Cloudera Manager does not store the credentials, after the installation / upgrade it will delete it. When you upgrade your ...


4

It should be safe enough to simply remove the Secondary Namenode role, and then add it back again onto a different node on the cluster. In the intermediate period, you may see a warning from Cloudera Manager that the role doesn't exist (which may cause long term issues with the namenode), but not having a Secondary doesn't put your data any more at risk.


3

The job of the 2NN is to read the changes on the HDFS filesystem and add them to the fsimage. This decreases NN startup time as during startup the NN reads the fsimage file and then applies all the interim log changes on top of it. The naming is somewhat unfortunate as it is indeed not a standby/backup NN but only a utility to increase NN performance. In ...


3

zless/zcat/zgrep are just shell wrappers that make gzip output the decompressed data to stdout. To do what you want, you'll just have to write a wrapper around the hadoop fs commands. Aside: The reason this probably didn't work for you is that you're missing an additional slash in your hdfs URI. You wrote: hadoop fs -cat hdfs://myfile.gz | zgrep "hi" ...


3

For the ambari-server-setup exec you define a path containing /usr/bin and /usr/sbin. Most likely you have your bash binary installed as /bin/bash, hence you will also need to include /bin in that path.


3

According to this documentation the /usr/lib/hbase/conf/hbase-site.xml file needs to be configured as follows: <configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> </configuration> to avoid HBase manages its own ZooKeeper. After configuring this, the ...


3

Let me jsut say: cheaper to buy better boxes. No joke - 8 cores, 10g ram is a low end server. There ARE 2 providers of "combined VM technology" but they are commercial and it COSTS. Like it is needed of you neded t o combine high end boxes. Single core license costs more than your outdated desktops combined, sadly. So, the asnwer is: no way. Check MOSIX ...


3

Sounds like you've got a mountain of knowledge to climb, I doubt you'll be able to adaquately design what you're looking for without learning all the technologies for a solid year or so first. That said, I'd forget using something fancy like Hadoop. You didn't mention what kind of storage you have, but I'd try to pull some sort of SAN together, possibly ...


3

If you're using the default Derby database for storing the Hive metadata, you must run Hive as a user with write permissions to the database file. You can use any database with a JDBC driver for storing the metadata; if you're running a dev environment it's fine but anything that gets more usage should move to a different database. Choices include MySQL, ...


3

You can do this by setting -Ddfs.block.size=something with your hadoop fs command. For example: hadoop fs -Ddfs.block.size=1048576 -put ganglia-3.2.0-1.src.rpm /home/hcoyote As you can see here, the block size changes to what you define on the command line (in my case, the default is 64MB, but I'm changing it down to 1MB here). :; hadoop fsck -blocks ...


3

Looks like I had mistakenly set fs.default.name to 9000. Changed it to 8020, now everything works fine. Thanks to mgorven for the help!


3

Multiple JobTracker can be useful in Multi-Cluster architecture. So the cluster level load can be distributed between the JobTrackers. In a single cluster, following could become issues. (a) If multiple JobTracker servers will share an HDFS cluster, each must have a different mapred.system.dir, or the JobTrackers will delete each other's job files. (b) ...


3

For an automatic deploy you'll need to dive into the world of Chef and Crowbar. Probably the best place to start is here. http://robhirschfeld.com/2011/11/29/hadoop-crowbar-released-to-open-source For a manual deploy, the folks over at Big Data Craft have implemented Hadoop on OpenStack’s Swift. You'll need to use their implementation of java-cloudfiles. ...


3

/sbin/init is always the first process launched on your system. If the binary doesn't exists, the system cannot start. I don't know what happened, but it seems it has been erased from your disk, the easiest way to repair it is to reinstall the entire system. If you did correctly install your system first, you'll just have to reinstall the root partition, and ...


3

I would recommend to keep your nodes as similar as possible. As you have found out, different setups for each node make life difficult. The clusters I currently run each have every box exactly the same, which means all my configuration for every node is the same. Configuration is stored on an NFS homedir. The machines are installed as standard CentOS, and ...



Only top voted, non community-wiki answers of a minimum length are eligible