Hot answers tagged

24

Straight from the horse's mouth: Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments ...


13

Use real server cases with good cooling/ventilation paths.


12

It would be incredibly bad. Hadoop eats heap like an elephant. The default heap size is 1000 MB but pretty much everybody increases it. 256 MB of RAM will not get you far, after the GPU and operating system take their toll, you'd probably have to restrict Java to 128 MB and the smallest jobs will run out of heap. The JVM is also very slow on ARM. Red Hat ...


11

Here's what I'd do. Get some large sheets of thin Lexan (or even foam board / corrugated plastic board - I know this as Correx, YMMV). Punch some holes through it, and bolt it to the holes on the uprights. Build about a foot or more higher on the top, cut out a hole and then mount a 12v automotive radiator fan on the top. You'll also want to build in ...


10

Thoughts: If you are asking these question on a public forum, hire experts to do it for you Consider Postgres and SQL Server which will scale to this volume too Do you need ACID? No = consider NoSQL Design and hardware matter more than the platform Don't virtualise or cut other hardware corners What is your RPO/RTO? Maintenance window? a.k.a are you really ...


9

Cloudera have some great videos that explain the principles behind Map Reduce and Hadoop. http://www.cloudera.com/hadoop-training-basic One of the core ideas behind MapReduce is that for large data sets you are going to be io bound on your disks, so in Hadoop HDFS gives you the ability to split things up between lots of nodes enabling parallel processing. ...


9

I feel quite strongly that two local servers, which everyone else binds to, is the right way to go. NTP is designed to work that way, and it minimises the load on the public/pool servers. I run an NTP pool server. Even in areas with well-populated pools the load is still significant (I'm running at an annualised average of 25 client requests per second, ...


7

My suggestion is to read the information provided. java.net.ConnectException: Connection refused; This generally means nothing is listening on the specified IP:Port. Note also this information is provided http://wiki.apache.org/hadoop/ConnectionRefused and it's first line You get a ConnectionRefused Exception when there is a machine at the ...


7

As the message tells you, any resource specification requires a name. In your specific case the following snippet should work: define hadoop_slave( $mem, $cpu, $dirs ) { hadoop_base { "${name}_hadoop_base": mem => $mem, cpu => $cpu, } hdfs_site { "${name}_hdfs_site": dirs => $dirs, } }


6

No. You can't suspend a job. There isn't. Slots are assigned for the duration of the tasks. You can decrease the priority in the scheduler which will allow some slots to be reclaimed as tasks complete.


6

Hadoop is a scalable data processing platform. MySQL is a relational database for realtime usage. HBase (running on HDFS --> Hadoop Distributed File System) is a BigTable implementation: Big tables but not relational! Can you map your application into these constraints? Only if you can do this mapping then the question of "will it perform" is a useful ...


6

The best thing you can do is get the disks replaced. The cost of disks won't weigh up against the cost of the cluster being down and your amount of work time being put in to fix the bad blocks. So even without a budget I would seriously try to convince your management.


5

CopyFromLocal does not have the ability to display the file copy progress. Alternatively, you could open another shell and run the $ watch hadoop fs -ls <filenameyouarecopying>. This will display the file and its size once every 2.0 seconds.


5

If you can I would look at utilizing Cloud Infrastructure Services like Amazon Web Services (AWS) Elastic Compute Cloud (EC2), at least until you determine that it makes sense to invest in your own hardware. It's easy to get caught up in buying the shiny gear (I have to resist daily). By trying before you buy in the cloud you can learn a lot and answer the ...


4

The Smart Array 410/i controller only allows 2 raid devices to be created without the optional memory card added to it. So if your hardware only has 2 drives, then it would be fine; you can just make it 2 raid0's. If you have more then 2 drives you will probably have to purchase Part #462974-001 - 256MB Battery Backed Write Cache BBWC memory module. If you ...


4

It is required during the installation and upgrade procedure when using the wizard in the Cloudera Manager. During normal cluster operation it is not needed. All communication is done via the locally running agents then. In fact Cloudera Manager does not store the credentials, after the installation / upgrade it will delete it. When you upgrade your ...


4

According to this documentation the /usr/lib/hbase/conf/hbase-site.xml file needs to be configured as follows: <configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> </configuration> to avoid HBase manages its own ZooKeeper. After configuring this, the ...


4

It should be safe enough to simply remove the Secondary Namenode role, and then add it back again onto a different node on the cluster. In the intermediate period, you may see a warning from Cloudera Manager that the role doesn't exist (which may cause long term issues with the namenode), but not having a Secondary doesn't put your data any more at risk.


4

Simply create the file and add your content in it. See this. Use gksudo gedit /etc/apt/apt.conf for graphical edit, or sudoedit /etc/apt/apt.conf for console edit. Regarding what you have to put in the configuration, you have to put the proxy address and its port (so fill in the blanks). If you do need a proxy, ofc.


4

zless/zcat/zgrep are just shell wrappers that make gzip output the decompressed data to stdout. To do what you want, you'll just have to write a wrapper around the hadoop fs commands. Aside: The reason this probably didn't work for you is that you're missing an additional slash in your hdfs URI. You wrote: hadoop fs -cat hdfs://myfile.gz | zgrep "hi" ...


3

This one has a simple solution. First, if you want to connect to port 8020 of your hadoop container, you should make sure that that port is exposed as well. Second, these containers each have their own loopback interface (localhost) and IP addresses. They are connected through the bridge network docker0 to the host's eth0 interface. So, you need to use the ...


3

With such a small environment, and it designated as just "testing", I would push NTP to the edge of the network. Run the NTP server on your switch or router if you can and configure it for 3 or more upstream time sources. Then, point all 7 nodes here. Likely, this is your only path out to the Internet so configuring additional time sources "inside" wouldn't ...


3

For the ambari-server-setup exec you define a path containing /usr/bin and /usr/sbin. Most likely you have your bash binary installed as /bin/bash, hence you will also need to include /bin in that path.


3

The job of the 2NN is to read the changes on the HDFS filesystem and add them to the fsimage. This decreases NN startup time as during startup the NN reads the fsimage file and then applies all the interim log changes on top of it. The naming is somewhat unfortunate as it is indeed not a standby/backup NN but only a utility to increase NN performance. In ...


3

Let me jsut say: cheaper to buy better boxes. No joke - 8 cores, 10g ram is a low end server. There ARE 2 providers of "combined VM technology" but they are commercial and it COSTS. Like it is needed of you neded t o combine high end boxes. Single core license costs more than your outdated desktops combined, sadly. So, the asnwer is: no way. Check MOSIX ...


3

Sounds like you've got a mountain of knowledge to climb, I doubt you'll be able to adaquately design what you're looking for without learning all the technologies for a solid year or so first. That said, I'd forget using something fancy like Hadoop. You didn't mention what kind of storage you have, but I'd try to pull some sort of SAN together, possibly ...


3

If you're using the default Derby database for storing the Hive metadata, you must run Hive as a user with write permissions to the database file. You can use any database with a JDBC driver for storing the metadata; if you're running a dev environment it's fine but anything that gets more usage should move to a different database. Choices include MySQL, ...


3

You can do this by setting -Ddfs.block.size=something with your hadoop fs command. For example: hadoop fs -Ddfs.block.size=1048576 -put ganglia-3.2.0-1.src.rpm /home/hcoyote As you can see here, the block size changes to what you define on the command line (in my case, the default is 64MB, but I'm changing it down to 1MB here). :; hadoop fsck -blocks ...


3

Check the hadoop-env.sh file in the Hadoop conf directory. Ensure the line exporting JAVA_HOME has been uncommented and set appropriately.


3

For an automatic deploy you'll need to dive into the world of Chef and Crowbar. Probably the best place to start is here. http://robhirschfeld.com/2011/11/29/hadoop-crowbar-released-to-open-source For a manual deploy, the folks over at Big Data Craft have implemented Hadoop on OpenStack’s Swift. You'll need to use their implementation of java-cloudfiles. ...



Only top voted, non community-wiki answers of a minimum length are eligible