Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My tomcat (6.0.32, Java Sun 1.6.0_22-b04 on Ubuntu 10.04) keeps crashing multiple times daily without any specific output in catalina.out. This usually happens on high load (see top output). Update: The pid-file is properly removed when this happens.

Update 2: No CATALINA_OPTS set, _JAVA_OPTS are:

export _JAVA_OPTIONS="-Xms128m -Xmx1024m -XX:MaxPermSize=512m \
-XX:MinHeapFreeRatio=20 \
-XX:MaxHeapFreeRatio=40 \
-XX:NewSize=10m \
-XX:MaxNewSize=10m \
-XX:SurvivorRatio=6 \
-XX:TargetSurvivorRatio=80 \
-XX:+CMSClassUnloadingEnabled \
-Djava.awt.headless=true \
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=37331 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=true \
-Djava.rmi.server.hostname=(myhostname) \
-Dcom.sun.management.jmxremote.password.file=/etc/java-6-sun/management/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/etc/java-6-sun/management/jmxremote.access"

Top:

top - 12:40:03 up 9 days, 12:15,  3 users,  load average: 30.00, 22.39, 21.91
Tasks:  89 total,   4 running,  85 sleeping,   0 stopped,   0 zombie
Cpu(s): 53.2%us,  9.7%sy,  0.0%ni, 34.7%id,  1.5%wa,  0.0%hi,  0.8%si,  0.0%st
Mem:   4194304k total,  3311304k used,   883000k free,        0k buffers
Swap:  4194304k total,        0k used,  4194304k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                       
25850 tomcat6   20   0 1981m 1.2g  11m S  161 29.6  11:41.56 java                                                                                          
12632 mysql     20   0  393m  97m 4452 S  141  2.4   1690:05 mysqld                                                                                        
14932 nobody    20   0  253m  44m 9152 R   56  1.1   3:26.57 php-cgi                                                                                       
 7011 nobody    20   0  241m  31m 9124 S   30  0.8   1:35.96 php-cgi                                                                                       
10093 nobody    20   0  228m  18m 8520 S   25  0.5   2:29.97 php-cgi                                                                                       
27071 nobody    20   0  237m  28m 8640 S   11  0.7   3:13.72 php-cgi                                                                                       
 3306 nobody    20   0  227m  16m 6736 R    7  0.4   2:29.83 php-cgi                                                                                       
 7756 nobody    20   0  261m  58m  15m R    5  1.4   2:22.33 php-cgi                                                                                       
 7129 www-data  20   0 3646m 7228 1896 S    2  0.2   0:36.65 nginx                                                                                         
 2657 nobody    20   0  228m  18m 8540 S    1  0.5   1:59.51 php-cgi                                                                                       
 7131 www-data  20   0 3645m 6464 1960 S    1  0.2   0:34.13 nginx                                                                                         
 7140 www-data  20   0 3652m  12m 1896 S    1  0.3   0:35.80 nginx                                                                                         
  619 nobody    20   0  231m  29m  15m S    0  0.7   2:33.46 php-cgi                                                                                       
16552 nobody    20   0  250m  41m 8784 S    0  1.0   2:48.12 php-cgi                                                                                       
17134 nobody    20   0  239m  37m  16m S    0  0.9   2:32.86 php-cgi                                                                                       
21004 nobody    20   0  243m  34m 8700 S    0  0.8   1:19.85 php-cgi                                                                                       
26105 root      20   0 19220 1392 1060 R    0  0.0   0:00.82 top                                                                                           
32430 nobody    20   0  256m  47m 9196 S    0  1.2   2:19.01 php-cgi                                                                                       
 314 nobody    20   0  256m  47m 8804 S    0  1.1   1:46.00 php-cgi                                                                                       
 2111 nobody    20   0  253m  44m 9196 S    0  1.1   3:01.14 php-cgi                                                                                       
 2142 root      20   0 26452 2564  868 S    0  0.1   0:00.56 screen                                                                                        
 2144 root      20   0 19484 2012 1368 S    0  0.0   0:00.00 bash                                                                                          
 2333 nobody    20   0  249m  41m 9160 S    0  1.0   1:10.33 php-cgi                                                                                       
 2552 root      20   0 19484 2260 1620 S    0  0.1   0:00.01 bash                                                                                          
 2587 nobody    20   0  258m  49m 9192 S    0  1.2   2:04.50 php-cgi                                                                                       
 2684 root      20   0  4092  652  540 S    0  0.0   0:00.00 xvfb-run                                                                                      
 2696 root      20   0 60720  13m 2352 S    0  0.3   0:09.12 Xvfb                                                                                          
 2759 root      20   0  617m  12m 4676 S    0  0.3   0:00.66 node                                                                                          
 3514 nobody    20   0  270m  61m 9216 S    0  1.5   3:13.69 php-cgi                                                                                       
 5270 root      20   0 25164 1324 1036 S    0  0.0   0:00.01 screen                                                                                        
 5402 nobody    20   0  227m  16m 8032 S    0  0.4   1:33.61 php-cgi                                                                                       
 5765 root      20   0 81180 3820 3028 S    0  0.1   0:00.31 sshd                                                                                          
 5798 nobody    20   0  242m  32m 9124 S    0  0.8   1:52.08 php-cgi                                                                                       
 5856 root      20   0 19496 2292 1636 S    0  0.1   0:00.03 bash                                                                                          
 6442 root      20   0 62332  20m 1960 S    0  0.5   0:30.58 mrtg                                                                                          
 7082 root      20   0 88992 1916 1636 S    0  0.0   0:00.00 PassengerWatchd                                                                          

I can't find any concrete reason for it, no Exceptions or messages of a shutdown in catalina.out (and no other logs in tomcat's log dir). I can start up the service and it will run for a few days or just minutes before dying again.

Is there somewhere else i could look for output? Could the kernel start killing threads due to a lack of ressources and by that bring the VM down?

share|improve this question
    
Do you have other log files, e. g. application-specific ones for Tomcat and what do your system log files say (/var/log/messages and the likes)? Your top says that there's still memory available, so I do not think the kernel starts killing processes. –  Axel Knauf Feb 21 '11 at 12:54
    
Are you by chance using the Oracle 11gR2 OCI JDBC driver? –  Corey S. Feb 21 '11 at 12:59
    
I can't find any other output. There is nothing in the os-logs and my webapps haven't got any logs of their own. –  Hendrik Feb 21 '11 at 13:35
    
@Corey: No, I am using MySQL, no Oracle server involved. –  Hendrik Feb 21 '11 at 14:10
    
Can you post more details such as your CATALINA_OPTS or other JVM parameters? –  Corey S. Feb 21 '11 at 14:40

2 Answers 2

If the process is dying it could be:

  1. JVM exited because of an unhandeled Java exception - you should see something in catalina.out or in the file used to redirect the STDOUT and STDERR - read as in application crash.
  2. OS send a signal that was not trapped by JVM - you should see something in STDOUT or STDERR that are redirected usually to catalina.out- read as in JVM crash.

Find where the STDOUT and STDERR is logged:

sudo -u tomcat6 ls -l /proc/$(pgrep java)/fd/{1,2}

Analise the java exception. If you do not find it, then search for a JVM creash.

Enable core dumps: http://en.linuxreviews.org/HOWTO_enable_core-dumps

Analise the core: http://www.freebsd.org/doc/en/books/developers-handbook/debugging.html#AEN1012

gdb /path.../java core
(gdb) bt

Analise the stack where JVM crashed.

EDIT: For high JVM load you can see which thread is generating the load. Use: top -Hp $(pgrep java) to find out the thread id that has the highest CPU usage. Then generate 2-3 thread dumps at 10 seconds apart with pkill -3 java. Convert to hex lowercase the PID of the thread from top output, then search for that number after nid= in catalina.out. That is the stack trace of the most expansive thread.

Also check if the JVM is not doing too much full GCs because the memory is not configured properly. See: http://java.sun.com/developer/technicalArticles/Programming/GCPortal/, http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html, http://www.oracle.com/technetwork/java/faq-140837.html

Some useful tools: http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html and https://visualvm.dev.java.net/

share|improve this answer
    
STDOUT and STDERR get logged to /var/log/tomcat6/catalina.out, where I would expect it, but where nothing gets logged. This is really weird. I don't know how to apply the core dump activation for Fedora on my Ubuntu server; I see if I can find a howto for that. –  Hendrik Feb 21 '11 at 15:41
    
Maybe they are rotated or overwritten at start-up time. –  Mircea Vutcovici Feb 21 '11 at 15:43
    
No, I can see the logs from the last start in it. –  Hendrik Feb 21 '11 at 15:45
    
How it went with the core dump/gdb and with thread dumps? –  Mircea Vutcovici Feb 21 '11 at 19:41
    
I'm still going through the dump, but it looks as if most expansive thread was in GC, for at least 30secs (3 dumps after 10secs each). Also, I see a lot of threads saying: java.lang.Thread.State: BLOCKED (on object monitor) - does this mean anything to you? –  Hendrik Feb 21 '11 at 21:26

Ubuntu will kill processes if the OS is running out of memory(at the OS level, i.e. the current JVM heap size, not the used size) If it were a standard JVM out of memory error then the process won't die, it will just stop doing meaningful work, and you would see the output in the catalina.out

See http://stackoverflow.com/questions/8146152/something-keeps-killing-my-java-process-on-ubuntu-anyone-know-why for more info.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.