1

I am running a few tomcat servers to which we deploy applications that our coders write and implement. Problem is that in some of our servers, especially the ones that have multiple requests, memory gets full very quickly and the server starts performing badly, if at all. I am new to this role and my predecessor mentions this is due to the code in the apps not being written correctly that it eats all the memory up... which is fair enough, but I am wondering if there is anything I can do to mitigate or eliminate this as currently the solution implemented until the code is improved is multiple weekly tomcat restarts, which in my opinion feels like an overkill! (no pun intended)

Below is the output of htop before tomcat needs to be killed and restarted (and that's another thing, most often tomcat can't be asked to politely quit, you must kill it -9, not sure if this is normal)

I've checked some resources but I wasn't able to find anything specific that could sort my problem out, so any good expertise would be welcomed!

I've included an image, as you can see the process seems to be repeated multiple times but it's not using over 300 gig memory like some people have been saying but only 7 gigs, not entire sure what that means.

Actually it could be a problem with htop as if you do a ps you can only see the below process:

root 5215 3.4 64.8 8310716 5301436 ? Sl Nov04 146:25 /usr/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.awt.headless=true -Xms5G -Xmx5G -XX:PermSize=512m -XX:MaxPermSize=512m -XX:NewSize=1G -XX:MaxNewSize=1G -Duser.langua

Anyway to get back to my point, it gets overloaded far too easy, any ways to prevent this in Tomcat version 7.0.28?

Here is the server.xml

<?xml version='1.0' encoding='utf-8'?>
<Server port="8105" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <GlobalNamingResources>


    <Environment name="databasePlaceHolder" type="java.lang.String" value="com_xxx_yyy_au"/>
    <Environment name="com.xxx.databasename" type="java.lang.String" value="com_xxx_yyy_au"/>
    <Environment name="com.xxx.JMS.url" type="java.lang.String" value="tcp://localhost:61616"/>
    <Environment name="remoteServerURL" type="java.lang.String" value="https://yyy.xxx.com/"/>

    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <Service name="Catalina">

    <Connector port="8109" protocol="AJP/1.3" redirectPort="0443" />

    <Engine name="Catalina" defaultHost="localhost">

      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

      </Host>
    </Engine>
  </Service>
</Server>

This is the setenv.sh file content:

JAVA_OPTS="-Djava.awt.headless=true -Xms5G -Xmx5G -XX:PermSize=512m -XX:MaxPermSize=512m -XX:NewSize=1G -XX:MaxNewSize=1G -Duser.language=en -Duser.region=GB"

JAVA_OPTS="${JAVA_OPTS} -XX:+UseParallelGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution"

This is the htop output, it's probably wrong

7
  • Why is that you have so many instances of tomcat running? Doesn't look normal to me. These are individual JVM processes each with MX size set to 5GB so no wonder you run out of memory quick. Please post your server.xml, catalina.properties, web.xml and startup.sh to help spot where the problem is. – Arul Selvan Nov 6 '15 at 2:00
  • Indeed, @ArulSelvan is right, counted them and you have 50 jvm's running each with 5G+ overhead memory usage. That means 250G + overhead, can go easily to 300GB. If your server does not have that much memory, the time when the applications start using memory for real you go swapping. – Fredi Nov 6 '15 at 15:09
  • That's what it shows there, but in reality it was only using 8 gigs memory, I don't know why they are repeated like that. Still, it should never get to 8 gigs, that's still mental! I want to know if there is anyway where Tomcat can be setup to recycle all processes, even if these go zombie. Will try and post more details such as the config files later. – Ulukai Nov 6 '15 at 21:38
  • I don't believe they are repeated. I see unique PIDs 17196, 17197, 17198... etc. As mentioned earlier, f you provide config files, startup script etc I listed above someone will be able to help you. Also, none of these processes can grow to 8G since the JVM is capped at 5G+overhead. – Arul Selvan Nov 7 '15 at 1:43
  • 1
    @Ulukai, as for htop, i think you have enabled thread info so each java entry you see is a thread of the same process. To verify this toggle "show threads" (Press "H" to toggle). – Fredi Nov 7 '15 at 17:42
1

As for htop being wrong, i think you have enabled thread info so each java entry you see is a thread of the same process. To verify this toggle "show threads" (Press "H" to toggle).

From the htop screenshot i notice your server has 8G of RAM, so having you capped the JVM to 5G + PermGen + some overhead, you should be ok assuming there are no other memory hungry processes running.

Next thing to check is the garbage collector, depending your java version there is a flag (at least in oracle java / openjdk) that makes the JVM to log each GC event, usually this: -verbose:gc -XX:+PrintGCDateStamps -Xloggc:SOMEFILENAME and check for even increasing GC activity, in case of a memory leak you'll see the garbage collection becoming more frequent as time passess until it uses all CPU trying to free memory without success, and you can get out of memory exceptions on your logs. At that point, you have to kill -9 the app as for you problem. But now you'll have a log of the GC activity post mortem to prove there are memory leaks or not.

Next, if you have more than one app deployed, try splitting the apps each in a single tomcat instance if possible. Or enable heap dump on out of memory.

As on what to do without the possibility of fixing the code, well, assuming there are memory leeks setup a monitoring on GC frequence, for example, if there are 3 Full GC attempts in a minute, automatically restart tomcat.

Ugly but if there are no other options it lets you sleep at night.

4
  • So GC happens automatically always then? if so, what triggers it? Can those triggers be altered? I don't have GC logs enabled (just posted setenv file) I assume I would need that to create the script you suggest, right? I just found out that GC can be done manually in probe, how could I do the same thing via the command line? Thank you! – Ulukai Nov 8 '15 at 17:59
  • So GC happens automatically always then? if so, what triggers it? Can those triggers be altered? ; here you cant do anything if you're not the programmer. – Fredi Nov 9 '15 at 2:43
  • Well I can force GC myself. I was thinking a solution would be scheduling it every night. Is this bad practise? – Ulukai Nov 9 '15 at 11:51
  • Well, if there is a memory leak forcing a GC does not help after a while, you have to restart the JVM – Fredi Nov 10 '15 at 14:53
1

Since I don't see the full command line args used to start the the JVM and not knowing the nature of the apps you deployed there, I can only guess that your apps are creating lot of 'long lived' objects which make their way into oldgen space and you are running out of memory there. Also, GC collection in oldgen space is pretty expensive and possibly your JVM at some point can't keep up with GC runs and coming to a grinding halt.

Having said that, I can suggest the following JVM tuning parameters.

Remove the two below:

-XX:NewSize=1G 
-XX:MaxNewSize=1G

And add the following:

-XX:+UseParallelOldGC
-XX:SurvivorRatio=10
-XX:NewRatio=2

If the problem is not solved, continue to increase the NewRatio to 3, 4, 5 and see when the JVM is stable enough to continue running w/ out any problem. Also, I am not sure why you would need 512M of permgen size i.e. -XX:PermSize=512m. Check with your app developers to see if they really need that much and reduce it if possible.

Also, when the problem happens, before you kill the process, run the following and post the output here that will give clues to people trying to help you here. (note: you have to run as root).

jmap -heap <pid_of_jvm>

PS: The htop output as explained by @Fredi is correct, it misleadingly labeled LWP thread IDs as PIDs.

2
  • I've added the setev.sh file at the end of the question. I don't really understand garbage collection that well, most of the documentation I found on it is scarce and very cryptic (as usual) I just know that when I click advise garbage collection in Probe, it takes a chunk out of used memory (I just discovered that), which is great and just what I need! Is there any way to schedule this so it does it once every at 2 am or something? What do the lines you gave me do exactly? – Ulukai Nov 8 '15 at 17:43
  • In short, the 3 options I mentioned tells JVM to use parallel GC collection in oldgen space and let the JVM decide the amount of heap partitioned between young gen, survivor, and oldgen space. So you need to remove the NewSize and MaxNewSize arguments. If you can run the jmap once you restart and once when you hit the problem and post the results, that would tell us how your JVM heap is used and someone may be able to suggest a better tuning option. Without the details of your JVM heap usage, there is not a whole lot anyone can do to help you. – Arul Selvan Nov 8 '15 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.