After a server bounce happens, and after around 40-45 days time period, we receive continuous “Committed Virtual Memory” alerts which indicates the usage of swap space in the magnitude of >4GB This also causes the application to perform very slowly and experience a number of stalled transactions. Server Setup:

4 Tomcat Servers (version 7.0.22) that are load balanced (not clustered) by 2 Apache Servers. And the Apache servers themselves supply static content and routing to these 4 tomcat servers.

Java Runtime Version: java version "1.6.0_30" Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode

Memory Startup Parameters:

MEMORY_OPTIONS="-Xms1024m -Xmx1024m -Xss192k -XX:MaxGCPauseMillis=500 -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled"

Monitoring – Wily monitoring is available in all the production servers that monitors key server parameters and sends out configurable alert emails based on pre defined settings.

Note: Each of the servers also has two other separate tomcat domains that run different applications

Investigated area:

  • There is no Heap Memory Leak and the GC is running fine without any issues over any period of time
  • The current busy thread count corresponds directly to the application usage – weekends and nights have lesser no. of threads compared to business hours
  • ThreadLocal uses a WeakReference internally. If the ThreadLocal is not strongly referenced, it will be garbage-collected, even though various threads have values stored via that ThreadLocal.
  • Additionally, ThreadLocal values are actually stored in the Thread; if a thread dies, all of the values associated with that thread through a ThreadLocal are collected.
  • If you have a ThreadLocal as a final class member, that's a strong reference, and it cannot be collected until the class is unloaded. But this is how any class member works, and isn't considered a memory leak.
  • The cited problem only comes into play when the value stored in a ThreadLocal strongly references that ThreadLocal—sort of a circular reference. In this case, the value (a SimpleDateFormat), has no backwards reference to the ThreadLocal. There's no memory leak in this code.

Can anyone please let me know what could be the cause of this and what to be monitored?

  • 1
    Exactly how much raw memory does this system contain? With 4 different java instances you are allocating 4GB immediately after all are started up. – mdpc Oct 29 '12 at 18:23
  • I dono about raw memory numbers but 4GB allocation is not happening immediately this grows for a month after the server start up – vinu Oct 30 '12 at 7:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.