Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two different tomcat 7 instances on two different servers (oracle jdk7) with nearly equal hardware configuration (both > 24 GB RAM). Both tomcat servers have the same configuration and the same web applications are deployed on these servers. The catalina opts are the following:

-XX:PermSize=128m -XX:MaxPermSize=512M -Xmx2048m -XX:+CMSIncrementalMode -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC

While running a load test (stressing a REST API with heavy parallel executed requests) one of the server throws java.lang.OutOfMemoryError: Java heap space (here is the stacktrace: http://pastebin.com/wuS1MVCC), the other server works just fine. I have no clue, why this happening. Has anyone ever faced a similar problem?

share|improve this question
    
Is the same OS and Application configuration for both systems? Both systems have a 64 bit OS? –  Sacx Nov 27 '12 at 12:57
    
Yes, same OS and similiar application configuration (only different database). Both system use Oracle 64Bit JVM. –  liecno Dec 3 '12 at 10:52
    
And the database of the server which throw the error, have more data? Did you tried to switch the databases between servers to see if the other server give the same error? –  Sacx Dec 3 '12 at 11:32
add comment

2 Answers

There is something different between the two systems otherwise the behaviour wouldn't be different. Note that too much time spent in GC for too little result can cause the error as well. Running close to the limits of the heap makes the GC have this type of problem.

Excessive GC Time and OutOfMemoryError

The concurrent collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line.

share|improve this answer
add comment

So what's happening is that one of your Tomcat JVMs is trying to exceed to 2048 MB heap that you allocated for it.

It just sounds like you're looking for a specific answer as much as a checklist of things to try, so here you go:

Running out of heap is either due to a memory leak, where you leak a little bit with each request, or in a load scenario it can be because you're just throwing more at the JVM than it can handle. You want to identify which of those is the problem, so start by looking at how you generate your load.

If the problem occurs only at a high level of parallel requests but never at a lower level, then you have a problem that the number of requests times the memory required to process each request is too big. You either need to make each request use less memory, or limit the concurrency somehow.

If the problem occurs after a certain number of requests have been processed regardless of the concurrency, then you have a memory leak. You need to find it and reclaim the memory.

In either scenario, you will be greatly aided by having a good heap memory analyzer. There are good commercial ones like YourKit Java Profiler, or free ones like Eclipse Memory Analyzer. Find a tool that works for you and learn how to use it to see what's taking up memory. Note that you don't necessarily need to use the tool to launch your program-- if you are running a load test on a server then you can use the jmap command-line tool from the JDK to capture a heap dump in a file, and then use your tool to analyze the dump file. The tool will show you what objects are taking up space in your heap.

share|improve this answer
    
This is a good answer, but does not help me analyzing the problem. The WebApp does not leak (already analyzed it). In fact, it is a heavy stress test with a lot of big requests, but it should be feasable. The interesting thing is, both server have a 2048MB Heap, both servers handle exactly the same requests (with equal content) both have the same webapps deployed and only one of them throws OOME. –  liecno Dec 3 '12 at 10:54
    
So there's got to be something different between the systems. When doing performance testing, even trivial-sounding differences matter. For example, if the latency to a database is a little higher, that can cause much higher memory utilization in some cases. How does memory usage compare between the two instances when the load test is running? Is one almost out of memory when the other goes out? If they are far apart, then capture a memory dump of both instances and see what objects the OOM instance is holding onto that the other one isn't. –  mark1ewis Dec 3 '12 at 21:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.