Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We run a cluster of ColdFusion 8 servers on top of the Sun JVM, version 1.6.0_11. Occasionally a server will simply stop responding to requests. We've determined so far that the JVM is locking up and that neither ColdFusion nor IIS are the root of the problem. I've included our JVM arguments below. Any additional JVM settings we should be looking at? Any tools to monitor, trace, or get metrics from inside the JVM so we can see what it's doing when the crashes occur? Any other JVM troubleshooting tips?

-server
-Xms1024m
-Xmx1024m
-Dsun.io.useCanonCaches=false
-XX:MaxPermSize=256m
-XX:PermSize=256m
-XX:+UseParallelGC
-Dsun.rmi.dgc.client.gcInterval=300000
-Dsun.rmi.dgc.server.gcInterval=300000
-Djmx.invoke.getters=true
share

locked by HopelessN00b Dec 5 at 7:48

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

    
What does the thread dump say. Without knowing your application I can suggest that one of the most likely causes of apparent lockup will be exhaustion of the database connection pool assigned to the application. This is generally pretty easy to detect from the thread dump –  Dave Cheney May 26 '09 at 15:49

5 Answers 5

up vote 1 down vote accepted

If you are on Linux you could use kill -3 to get a thread dump and look for deadlocks using using Thread Dump Analyzer.

VisualVM now ships as part of the JDK and can be used monitor memory usage, threads and also includes a profiler.

jmap can be used to obtain heap histograms and heap dumps from Java processes. You could then use a tool like Eclipse Memory Analyzer to investigate it.

As another answer mentioned you could look into JMX and JConsole. For each service in our application we have a MBean so we can view keys stats such as queue sizes to check if anything seems out of the ordinary.

share
    
The very recent JDK versions have a better edition of the VisualVM than the older ones. Take care that you have a recent JDK. –  djangofan Jun 9 '11 at 17:24

This is a standard option for my config:

-XX:+HeapDumpOnOutOfMemoryError

I've also have SNMP/RRD monitoring of basic JVM health characteristics (like heap size, thread count)...there are more.

Then, there is the entire world of tools like jconsole...

share
    
Problem is that this will sometimes consume minutes of time while the server is offline. If you monitor the server and restart it if it stops responding then you might kill the JVM dump before its finished dumping. –  djangofan Jun 9 '11 at 17:25

You need to look into JMX monitoring - it can give you good insight as to what's going on inside the JVM.

To start, add -Dcom.sun.management.jmxremote.port=portNun to the list above. Then, on another machine, run jconsole and tell it to connect to the machine and port you specified above. Poke around at will and see what threads are running and what's going on after it's stopped responding to requests.

share

jvmstat has been helpful to me in the past, though I don't think I used it with CF 8/JRE 1.6. It's got nice displays of what's in each generation so you can see if some of your memory variables are causing the problem.

share

you can use jstat to see the heap usage.

jstat -gc -h10 -t 1000

share
    
I love that tool. Also, jhat.exe . –  djangofan Jun 9 '11 at 17:26

Not the answer you're looking for? Browse other questions tagged or ask your own question.