2
votes

We run a cluster of ColdFusion 8 servers on top of the Sun JVM, version 1.6.0_11. Occasionally a server will simply stop responding to requests. We've determined so far that the JVM is locking up and that neither ColdFusion nor IIS are the root of the problem. I've included our JVM arguments below. Any additional JVM settings we should be looking at? Any tools to monitor, trace, or get metrics from inside the JVM so we can see what it's doing when the crashes occur? Any other JVM troubleshooting tips?

-server
-Xms1024m
-Xmx1024m
-Dsun.io.useCanonCaches=false
-XX:MaxPermSize=256m
-XX:PermSize=256m
-XX:+UseParallelGC
-Dsun.rmi.dgc.client.gcInterval=300000
-Dsun.rmi.dgc.server.gcInterval=300000
-Djmx.invoke.getters=true
1
  • What does the thread dump say. Without knowing your application I can suggest that one of the most likely causes of apparent lockup will be exhaustion of the database connection pool assigned to the application. This is generally pretty easy to detect from the thread dump May 26, 2009 at 15:49

5 Answers 5

1
vote

If you are on Linux you could use kill -3 to get a thread dump and look for deadlocks using using Thread Dump Analyzer.

VisualVM now ships as part of the JDK and can be used monitor memory usage, threads and also includes a profiler.

jmap can be used to obtain heap histograms and heap dumps from Java processes. You could then use a tool like Eclipse Memory Analyzer to investigate it.

As another answer mentioned you could look into JMX and JConsole. For each service in our application we have a MBean so we can view keys stats such as queue sizes to check if anything seems out of the ordinary.

1
  • The very recent JDK versions have a better edition of the VisualVM than the older ones. Take care that you have a recent JDK.
    – djangofan
    Jun 9, 2011 at 17:24
2
votes

This is a standard option for my config:

-XX:+HeapDumpOnOutOfMemoryError 

I've also have SNMP/RRD monitoring of basic JVM health characteristics (like heap size, thread count)...there are more.

Then, there is the entire world of tools like jconsole...

1
  • Problem is that this will sometimes consume minutes of time while the server is offline. If you monitor the server and restart it if it stops responding then you might kill the JVM dump before its finished dumping.
    – djangofan
    Jun 9, 2011 at 17:25
0
votes

You need to look into JMX monitoring - it can give you good insight as to what's going on inside the JVM.

To start, add -Dcom.sun.management.jmxremote.port=portNun to the list above. Then, on another machine, run jconsole and tell it to connect to the machine and port you specified above. Poke around at will and see what threads are running and what's going on after it's stopped responding to requests.

0
votes

jvmstat has been helpful to me in the past, though I don't think I used it with CF 8/JRE 1.6. It's got nice displays of what's in each generation so you can see if some of your memory variables are causing the problem.

0
votes

you can use jstat to see the heap usage.

jstat -gc -h10 -t 1000

1
  • I love that tool. Also, jhat.exe .
    – djangofan
    Jun 9, 2011 at 17:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .