Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

On several machines I have a problem with a process that is using 100% time of a CPU.

USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND

root 85740 100.0 0.2 34668 6352 ?? R 12:06AM 472:20.86 /usr/bin/javad (perl5.12.4)

At this moment, this process is running almost for over 8 hours:

# ps -p 85740 -o etime=

08:05:25

# ps -auxeww shows full command as: root 29323 99.0 0.2 32292 6692 ?? R 11:21PM 596:50.14 SUDO_COMMAND=/usr/local/bin/bash LOGNAME=root MAIL=/var/mail/root PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/fgz/bin SUDO_GID=1004 OLDPWD=/usr/local PWD=/usr/local/jboss4/bin _=./start.sh TERM=xterm USER=root HOME=/root SHELL=/usr/local/bin/bash SUDO_USER=fgz SUDO_UID=1004 USERNAME=root JBOSS_HOME=/usr/local/jboss4 SHLVL=1 /usr/bin/javad (perl5.12.4)

On this particular machine we have JBoss AS installed (and also on others).

I would appreciate any hints how to source which starts this process. In /etc/crontab or crontab -l I don't have any actions that would start perl.

And what is interesting:

# ls -al /usr/bin/javad

ls: /usr/bin/javad: No such file or directory

While process is still running.

Other Java processes that are running:

USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND

root 1322 0.0 0.0 14688 1344 0- I 24Jun13 0:00.00 /bin/sh ./run.sh -b 10.0.5.9

root 1334 0.0 11.8 2465148 493584 0- I 24Jun13 40:40.66 [java]

Where run.sh is a JBoss Bootstrap Script.

share|improve this question
    
The user running the command is fgz, would it be possible to ask them what they did? –  Jenny D Jul 11 '13 at 9:03
    
Dump the stack of the Java process. There's a high probability it is stuck in a while { ... } loop with no valid exit condition. If you can repeatedly dump the stack and find it staying in the same routine then you can tell the developer they have a problem in that particular function. –  PP. Jul 25 '13 at 8:45
    
@PP. is this a proper way to do it: "kill -SIGQUIT process_id will cause a core dump from a running process (assuming that resource limits allow it). Or see man 3 abort for causing a program to dump itself."? I would appreciate any hint how to do it. –  Mark Zakred Jul 25 '13 at 9:27

2 Answers 2

Whenever I see perl consuming 100% CPU on a machine running JBoss, I'm willing to bet that the machine has been infected by a worm.

Just this morning I also discovered a non-existent /usr/bin/javad running on a Linux box where JBoss AS is installed. Investigation revealed that there was a security hole - someone from Romania had accessed /invoker/JMXInvokerServlet and managed to deploy a 'web application' (yeah!) on the server. You should check if this 'invoker' is accessible on your server. If it is, the next question would be, is it really necessary? In my case I just removed it (by removing the $JBOSS_HOME/server/default/deploy/http-invoker.sar directory), restarted JBoss and verified that the web application is still running.

share|improve this answer

Here's an article on monitoring Java processes: http://www.ibm.com/developerworks/library/j-5things8/

The most useful tool is jstack. If it is installed on your system then use it to show what the busy process is doing. I should mention that jstack is a command-line tool. You use it along with the process ID of the busy Java process. It should give a stack dump of every thread in that process. Most threads should be waiting for a synchronisation object (indicating that they are not busy, not using any CPU).

Failing the existence of jstack you can send an appropriate signal to the process - but finding where the stack is dumped can be a challenge. See http://stackoverflow.com/questions/3734696/how-to-get-stack-trace-of-a-thread for details.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.