let me first describe what we do:
Our Tomcat hosts aprox. 2300 Jsp Files. These files are generate every night (even if they do not change) from a Content Management Server and are copied the tomcat directory. From time to time the tomcat stops working because no heap space was available.
The following part of the Heap keeps increasing:
concurrent mark-sweep generation: capacity = 8018026496 (7646.5859375MB) used = 5234557528 (4992.06307220459MB) free = 2783468968 (2654.52286529541MB) 65.28486193717862% used
After a restart of the Server, this grows to around 30% in the course of the day and jumps to 80% in the night (one time at 2:15 am and one time at 4:00 am). The deployment of the new JSPs has been done at 11pm the day before.
My approach for detecting the problem:
I triggere 2 heapdumps on a still running system. These were taken 24h apart. I analyzed these individually and compared them with each other.
By looking at the retained size:
Are the top candidates.
These Heapdumps were also feed into Eclispe Memory analyzer. This tool gave me the hint:
One instance of "org.apache.jasper.servlet.JspServlet" loaded by "org.apache.catalina.loader.StandardClassLoader @ 0x594cc5300" occupies 962.914.048 (88,21 %) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment" loaded by "".
This led me to the following information:
Both observations led me to believe, that the cached JSPs are polluting the heap and I took measure by reconfiguring the tomcat server.
I set the parameter maxLoadedJsps to 2000 and restarted the server. This was kind of a remedy. Still jumping at 4am but now to 60% only. So my change did improve something, but considering the JVM has 16 GB Heap available, this seems a bit unsatisfying. For one it still climbs slowly and I wanted to reduce the Heapspace for the Jvm if possible.
I'm looking for a little bit more insight into the connection between my reading of the heapdumps and the behavior of tomcat/jasper?
I think for tomcat all newly deployed JSP have to be cached besides the obsolete once's he is caching already. And I think this caching takes place when maybe Google is crawling the website in the nite. Putting almost all JSP in the Cache twice.
Can (maybe) someone confirm that my assumptions are not totally off and if that is the case guide me to a better explanation. Since it contains confidential data, I am not able to publicly share the heapdumps. But maybe some has some hints for better analyzing them.
Thanks for your help. Thomas