Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I want to run nutch on the linux kernel,I have loged in as a root user, I have setted all the environment variable and nutch file setting. I have created a url.txt file which content the url to crawl, When i am trying to run nutch using following command,

bin/nutch crawl urls -dir pra

it generates following exception.

crawl started in: pra
rootUrlDir = urls
threads = 10
depth = 5
Injector: starting
Injector: crawlDb: pra/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.io.IOException: Failed to get the current user's information.
        at org.apache.hadoop.mapred.JobClient.getUGI(JobClient.java:717)
        at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:592)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:788)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
        at org.apache.nutch.crawl.Injector.inject(Injector.java:160)
        at org.apache.nutch.crawl.Crawl.main(Crawl.java:113)
Caused by: javax.security.auth.login.LoginException: Login failed: Cannot run program "whoami": java.io.IOException: error=12, Cannot allocate memory
        at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
        at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
        at org.apache.hadoop.mapred.JobClient.getUGI(JobClient.java:715)
        ... 5 more

Server has enough space to run any java application.I have attached the statics..

            total       used       free  
Mem:        524320     194632     329688 
-/+ buffers/cache:     194632     329688
Swap:      2475680          0    2475680
Total:     3000000     194632    2805368

Is it sufficient memory space for nutch? Please some one help me ,I am new with linux kernel and nutch. Thanks in Advance.

share|improve this question

migrated from stackoverflow.com Jan 14 '10 at 13:27

This question came from our site for professional and enthusiast programmers.

3  
It's a really bad idea to run things as root that don't need to be run as root. Not only is it a security nightmare, but it can interfere with things that do need to be run as root, like servers. –  Paul Tomblin Dec 28 '09 at 14:25

4 Answers 4

up vote 2 down vote accepted

Read the output:

Cannot run program "whoami": java.io.IOException: error=12, Cannot allocate memory

Looks like you don't have enough RAM or no swap file/partition.

share|improve this answer
    
There is enough memory on server. total used free shared buffers cached Mem: 524320 213700 310620 0 0 0 -/+ buffers/cache: 213700 310620 Swap: 2475680 0 2475680 Total: 3000000 213700 2786300 Can you please suggest me? –  prakash.panjwani Jan 8 '10 at 18:59
    
At the time when whoami is called, there is no memory left -> You must run the stats command while the tool is running, not afterwards. –  Aaron Digulla Jan 10 '10 at 14:05

It is possible that your server has disabled /proc/sys/vm/overcommit_memory. Without overcommit, a "fork" system call requires that your server have enough RAM or swap for a complete second copy of the Java process. This may be a lot of RAM.

share|improve this answer

Calls to executables (like whoami) in Java require making an entire copy of the Java process first. You will want to drop your maximum heap size (-Xmx256m) to where you may have two copies in RAM at the same time.

share|improve this answer

In 32 bit installation of an Operating System the JVM(Java Virtual Machine) can not handle memory larger that 4GB. If you want to use JVM to take more than 4GB then you have to use 64bit version of the JVM which also means that the Operating System should also be 64 bit version.
I presume that is why you are getting that error. You have 5GB memory and that could be the problem. You should either tell your application to only use 75% of the available memory or try reducing the RAM to 4GB and checking. I had the same issue in Zimbra Messaging solution which uses Java for the Web interface.

share|improve this answer
1  
he has 512M of ram, not 5G –  Justin Jan 16 '10 at 11:25

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.