I am working on a virtual server from a local online hosting service where I would like to run an apache karaf instance.
During start-up of the server, I get out-of-memory exceptions:
Exception in thread "Karaf local console user karaf" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
(see below for more details)
I have no idea how this is happening as the server has 2GB of memory available.
When logging in with another ssh-shell I sometimes get the error message
/usr/bin/lesspipe: 1: /usr/bin/lesspipe: Cannot fork
QUESTION: why won't karaf start up completely? Which steps would you take to analyze the problem in more detail?
Thank you very much for your support!
Here some more information:
the karaf setenv file
export JAVA_MIN_MEM=256M # Minimum memory for the JVM
export JAVA_MAX_MEM=1024M # Maximum memory for the JVM
export JAVA_PERM_MEM=256M # Minimum perm memory for the JVM
export JAVA_MAX_PERM_MEM=1024M # Maximum perm memory for the JVM
output of free -m
total used free shared buff/cache available
Mem: 2048 223 1363 0 461 1824
Swap: 0 0 0
There seems to be something left, even after the karaf server has stopped working:
top - 16:04:28 up 2 days, 1:09, 2 users, load average: 0.00, 0.01, 0.05
Tasks: 26 total, 1 running, 25 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.4 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 2097152 total, 1445120 free, 216712 used, 435320 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 1880340 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8439 root 20 0 97176 4180 3240 S 0.3 0.2 0:00.34 sshd
8793 karaf 20 0 3187104 192736 15088 S 0.3 9.2 0:08.82 java
cat /proc/sys/kernel/threads-max
3092863
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1546431
max locked memory (kbytes, -l) 16384
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 62987
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
2810691: kmemsize 73211904 74088448 9223372036854775807 9223372036854775807 0
lockedpages 0 0 9223372036854775807 9223372036854775807 0
privvmpages 120109 126756 9223372036854775807 9223372036854775807 0
shmpages 277 2064 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numproc 78 78 400 400 0
physpages 214852 219387 524288 524288 0
vmguarpages 0 0 9223372036854775807 9223372036854775807 0
oomguarpages 216059 220594 0 0 0
numtcpsock 0 0 9223372036854775807 9223372036854775807 0
numflock 7 15 9223372036854775807 9223372036854775807 0
numpty 4 5 9223372036854775807 9223372036854775807 0
numsiginfo 0 123 9223372036854775807 9223372036854775807 0
tcpsndbuf 0 0 9223372036854775807 9223372036854775807 0
tcprcvbuf 0 0 9223372036854775807 9223372036854775807 0
othersockbuf 0 0 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 0 9223372036854775807 9223372036854775807 0
numothersock 0 0 9223372036854775807 9223372036854775807 0
dcachesize 55529472 55709696 9223372036854775807 9223372036854775807 0
numfile 966 1395 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numiptent 36 36 2000 2000 0
df -h
/dev/ploop11992p1 51G 2.1G 46G 5% /
none 1.0G 0 1.0G 0% /sys/fs/cgroup
none 1.0G 0 1.0G 0% /dev
tmpfs 1.0G 0 1.0G 0% /dev/shm
tmpfs 1.0G 1.1M 1023M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
none 1.0G 0 1.0G 0% /run/shm
Content of memory.log
Number of objects waiting finalization: 0
Heap:
Init: 268435456
User: 63690360
Committed: 259588096
Max: 1037959168
Non-Heap:
Init: 7667712
User: 60647112
Committed: 68009984
Max: -1
Long error message
Apache Karaf (4.2.5)
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown Karaf.
[7,773s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[7,794s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Exception in thread "Karaf local console user karaf" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:803)
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at org.apache.felix.gogo.runtime.CommandSessionImpl$JobImpl.start(CommandSessionImpl.java:799)
at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:300)
at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:169)
at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:149)
at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:179)
at org.apache.karaf.shell.impl.console.ConsoleSessionImpl.doExecuteScript(ConsoleSessionImpl.java:578)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at org.apache.karaf.shell.impl.console.ConsoleSessionImpl.executeScript(ConsoleSessionImpl.java:570)
at org.apache.karaf.shell.impl.console.ConsoleSessionImpl.run(ConsoleSessionImpl.java:401)
at java.base/java.lang.Thread.run(Thread.java:834)
[76,701s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGWINCH to handler- the VM may need to be forcibly terminated
[78,236s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGWINCH to handler- the VM may need to be forcibly terminated
