3

I’m trying to start a process on Red Hat Enterprise Linux Server release 6.5 (Santiago), which is failing because it is not able to allocate sufficient memory.

# There is insufficient memory for the Java Runtime Environment to continue. 
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:

The problem is not with the Java application itself - I get the error too when I simply run:

java -version

Looking at memory consumption:

free -m

             total       used       free     shared    buffers     cached
Mem:         32069      31276        792          0        556      16948
-/+ buffers/cache:      13771      18297
Swap:         9325         38       9287

So Mem is reporting that 31 out of 32 Gb of memory is being used, but there should be 18Gb of memory available that is being used by the cache. Should the OS not free up some of this when a process requests memory?

The details of the error dump is

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (gcTaskThread.cpp:46), pid=40816, tid=140071992215296
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed oops)

---------------  T H R E A D  ---------------

Current thread (0x00007f6508006800):  JavaThread "Unknown thread" [_thread_in_vm, id=40817,              stack(0x00007f650d46c000,0x00007f650d56d000)]

Stack: [0x00007f650d46c000,0x00007f650d56d000],  sp=0x00007f650d56b7c0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x85ebd5]  VMError::report_and_die()+0x265
V  [libjvm.so+0x3e41b8]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x68
V  [libjvm.so+0x466ada]  GCTaskThread::GCTaskThread(GCTaskManager*, unsigned, unsigned)+0x13a
V  [libjvm.so+0x4655de]  GCTaskManager::initialize()+0x21e
V  [libjvm.so+0x465373]  GCTaskManager::GCTaskManager(unsigned)+0x13
V  [libjvm.so+0x722cad]  ParallelScavengeHeap::initialize()+0x4dd
V  [libjvm.so+0x836549]  Universe::initialize_heap()+0xa9
V  [libjvm.so+0x8360ea]  universe_init()+0x7a
V  [libjvm.so+0x4ac53b]  init_globals()+0x4b
V  [libjvm.so+0x81cc74]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x214
V  [libjvm.so+0x51a7b0]  JNI_CreateJavaVM+0x80
1

Have a look into the jvm min max memory, do you use any web server? Look into the number of treads, memory settings for it- for Tomcat most of them are @server.xml. Look into Java Mission Control http://www.oracle.com/technetwork/java/javaseproducts/mission-control/java-mission-control-1998576.html to get extra insight. Linux usually keeps more memory to be ready when requested. For Java on Linux only something like JMC can help you get an idea on what's going on. Have a look in the standard output from the JVM.

  • Nice link and good info for general Java memory issues, but I'm more curious here about why the OS does not deallocate cache memory for a simple "java -version". – notdazedbutconfused Jan 28 '15 at 12:11
  • This is more related to GC you can have a look here cubrid.org/blog/dev-platform/… keeps the RAM ready, the JVM is requesting it and also handling it. Look at XX:PermSize and -XX:MaxPermSize and various memory settings – Alex H Jan 28 '15 at 12:36
  • Thanks Alex, I think one of my team mates actually found the issue. Useful links nonetheless! – notdazedbutconfused Jan 28 '15 at 12:41
  • Great, that's what matters anyhow ;) – Alex H Jan 28 '15 at 14:59
0

Seems like the issue has nothing to do with memory, but with process limits. The user I'm running the process as had a really low process limit set to 1024.

ulimit -u
1024

and the thread count of all the proceses run by this user is close to that:

ps -eLf | grep 'myuser' | wc -l
1022
-2

"free -g" will give you the current memory usage, if you are not having enough memory, then increase the physical RAM or increase the swap space.

if you have enough memory, then the next step would be increasing the max user process count.

change ulimit configuration(add below lines at the end)

vi /etc/security/limits.conf

* soft nproc 65535

* hard nproc 65535

* soft nofile 65535

* hard nofile 65535

vi /etc/security/limits.d/90-nproc.conf * soft nproc 65535

* hard nproc 65535

* soft nofile 65535

* hard nofile 65535

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.