Tomcat's falling again and again

Question

We are a happy group of developers in venezuela who developed a tomcat5.5+java+spring+hibernate+mysql+jstl+(flex) application. Now we installed them to customers and make them happy, each of them in their own domain. We use Linode as hosting provider, and cpanel's WHM software to help us create accounts rapidly, and mantain the mail server's and stuff. And give users acces to their mail accounts.

So, we were all getting along until my tomcat started to stop working from time to time, gave memory error, restarted the tomcat and all was fine again. Got more clients, bought more hardware until hardware did not make any difference, learned about heap size, now i give more heap size to tomcat, now the whole server falls down from time to time. Connected Jconsole, i can se peaks in memory each 10 minutes, some times less frequency. Te peak is higher each iteration until it breaks. Need tips on how to scale this kind of application, so i ask for the big guys (you guys) for any tips, directions, anything, for scaling my architecture. Thank you!

Answer 1

If the whole server falls you are posibly allocating too much memory for the JVM. You must leave some room for the os, network connections and buffers. On the JVM side, if your memory usage is continuously growing you have some kind of leak or are keeping sessions too much time, or simply allocated memory is not enough for this user number. Take some metrics about memory usage per session.

The jvm behaviour is to allocate memory as long as it free, when it cannot find free memory any more then it garbage collects (partial gc or full gc depending on memory pressure), you should check the lower point after garbage collections to see if its going higher with each gc. Also check gc times, if your application is spending too much time in gc and you cannot give it more memory is time to scale out and load balance.