How to reliably take Java Heap Dumps?

Question

My team is running into difficulties when trying to take good heap dumps triggered by OutOfMemoryErrors. For specific reasons we are currently taking the dumps with jmap called from a bash script instead of using the HeapDumpOnOutOfMemoryError flag. We're using a 64-bit 1.6 JVM with a heap size around 3 GB. Our heap dumps fail 90% of the time (guesstimate).

Is there anything we can do to improve our odds of getting a clean heap dump we can use to troubleshoot memory problems? I have read that jmap had major issues in Java 1.4 but that those issues should be mostly addressed now.

Answer 1

we have a JSP that queries ManagementFactory.getThreadMXBean() and produces a report. May not be useful when the app has crashed, but if you poll every minute or so, you'll get an idea of what's happening.

More info here.

Answer 2

you could monitor your application via jmx from the outside. when you know some metrics which indicate an upcoming OutOfMemory, you could trigger a jmap run before the exception is thrown.

Answer 3

Which is your operating system? (I can't add comments).

For Solaris we get better results first forcing a core dump (gcore <pid>) and then attaching jmap to the core dump file (jmap -heap:format=b <path to java bin> <path to core>)

gcore is a *nix utility to generate an image of a running program. See link.

Answer 4

Thanks to you all for your suggestions.

What we wound up doing is writing a script to actively monitor the garbage collection logs. In our experience, back-to-back Full GC's almost always precede an OOM, so our script detects this event, gracefully removes the server from the load balancing pool, and forces the heap dump. This has greatly increased our effectiveness.