Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My team is running into difficulties when trying to take good heap dumps triggered by OutOfMemoryErrors. For specific reasons we are currently taking the dumps with jmap called from a bash script instead of using the HeapDumpOnOutOfMemoryError flag. We're using a 64-bit 1.6 JVM with a heap size around 3 GB. Our heap dumps fail 90% of the time (guesstimate).

Is there anything we can do to improve our odds of getting a clean heap dump we can use to troubleshoot memory problems? I have read that jmap had major issues in Java 1.4 but that those issues should be mostly addressed now.

share|improve this question
4  
I nominate this question for "most unintentionally disgusting-sounding". –  phoebus Feb 6 '10 at 16:04
    
Hah- I thought about making it intentionally disgusting-sounding but I'm new here and I wasn't sure how the community would take that :). –  karlcyr Feb 6 '10 at 16:21
add comment

5 Answers 5

Which is your operating system? (I can't add comments).

For Solaris we get better results first forcing a core dump (gcore <pid>) and then attaching jmap to the core dump file (jmap -heap:format=b <path to java bin> <path to core>)

gcore is a *nix utility to generate an image of a running program. See link.

share|improve this answer
    
tried it with gdb on linux and it works great. –  Christian Mar 25 '10 at 10:17
    
which JDK has "gcore" ? mine, Sun 32 bit jdk for linux 1.6.0.20 doesnt have it. –  djangofan Mar 17 '11 at 23:27
    
Edited with gcore clarification. –  fglez Mar 18 '11 at 12:49
add comment

we have a JSP that queries ManagementFactory.getThreadMXBean() and produces a report. May not be useful when the app has crashed, but if you poll every minute or so, you'll get an idea of what's happening.

More info here.

share|improve this answer
add comment

you could monitor your application via jmx from the outside. when you know some metrics which indicate an upcoming OutOfMemory, you could trigger a jmap run before the exception is thrown.

share|improve this answer
    
Thanks Christian- is jmap more likely to be reliable before the error is thrown? –  karlcyr Feb 6 '10 at 21:22
    
jmap will still need some time to get you a heap dump. but you will get a full heapdump as long as your jvm/tomcat is mostly responsible. –  Christian Feb 7 '10 at 10:05
    
I think the cleanest and easiest tool to do this is "Visual VM". It might be out of scope but making a custom plugin for VisualVM that detects the condition and takes the automatic dump from within VisualVm would be awesome IMHO. –  djangofan Apr 7 '10 at 17:53
add comment

Thanks to you all for your suggestions.

What we wound up doing is writing a script to actively monitor the garbage collection logs. In our experience, back-to-back Full GC's almost always precede an OOM, so our script detects this event, gracefully removes the server from the load balancing pool, and forces the heap dump. This has greatly increased our effectiveness.

share|improve this answer
add comment

This is a fairly old question, but i will put answer with the hope that somebody may find this useful.

jmap has a -F option (force). This has proven to not work that well in the past for me. If you are to use the -F option, I would recommend that you also specify the java.io.tmp directory as a part of the jmap command. There was an issue with JVM version 1.6.22 where the jmap utility did not work properly because of a temp directory setting.

You can also try to take a core dump via gdb. Once you have the core, jmap can convert the core into a heap dump.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.