0

We upgraded from Ubuntu 16LTS to Ubuntu 18LTS a couple of days ago and since then have had Apache Tomcat crash completely twice (once per day, about 25 hrs apart). The java process stops running.

One line is added to the log file at this point:

free(): corrupted unsorted chunks

(that's it - no timestamp)

It doesn't seem to correlate to any particular action happening on the server at those times, at least it's hard to tell with only two instances, but my suspicion is it may have something to do with garbage collection. That's because the server monitoring shows memory usage for the java process drop from 7.90GB to 0.93GB within a minute at that point (in actual fact the java process quits so it's possible that smaller amount is after I manually restarted tomcat). The server's max memory setting -Xmmx is set to 8GB and gradually before the crash it creeped up to just below that through the course of the day.

Furthermore, the very first lines of the tomcat log are

NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

In /etc/default/tomcat8 I have the line

JAVA_OPTS="-Djava.awt.headless=true -Xmx8g -XX:+UseConcMarkSweepGC"

That UseConcMarkSweepGC option was added back on Ubuntu 16 where tomcat7 was used and I seem to remember it was recommended. In fact the config. file still says

'Use "-XX:+UseConcMarkSweepGC" to enable the CMS garbage collector (improved response time)

Anyway I've now removed it in the hope that that was the issue. We will see if there's a crash tomorrow but in the meantime does that seem reasonable? Can anyone suggest any steps to validate this hypothesis or debug further, or any other ideas for what might be happening?

1
  • +UseConcMarkSweepGC wasn't the issue, it's still crashing (slightly less often), so I'm still looking for a solution! – Oliver Kohll Jun 21 '18 at 16:21
0

In case anyone else has this issue, thanks to the helpful java-core-libs mailing list, I found the cause was the use of the APR/native libraries in Tomcat. Uninstalling these (apt-get remove libapr1) and removing configuration references from server.xml resolved the issue

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.