0

I have a standalone application where I am trying to create 5 threads for each thread I am creating a mongoclient and persisting the details in MongoDB and then closing the mongoclient.

When I start my mongo service it is running but as soon as I start my java application after some time my mongo service is getting killed and I can see the below error in syslog.

Out of memory: Kill process 12715 (mongod) score 433 or sacrifice child\\
kernel: [2946780.340246] Killed process 12715 (mongod) total-vm:6646800kB, anon-rss:6411432kB, file-rss:0kB

I am using Linux server it has 10GB RAM. As far as I know we can do thousands of transactions in MongoDB. And before I start I can see free memory after some time I am getting the above error.

Not sure exactly what to do, Do we need to set anything in linux server to handle multiple mongo requests? or do I need to increase the limit of mongo service any where?

Can any one please suggest me on this?

Thanks in Advance.

o/p of free -m before and after execution of my java application are as follows:

Before:

             total       used       free     shared    buffers     cached
Mem:         14032       7646       6385         54          4        153
-/+ buffers/cache:       7489       6542
Swap:            0          0          0

After:

             total       used       free     shared    buffers     cached
Mem:         14032      13845        186         54        106       4131
-/+ buffers/cache:       9607       4425
Swap:            0          0          0

The below are the details from mongod.conf

# mongod.conf

storage:
  dbPath: /xxx/xxx 
  journal:
    enabled: true

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

net:
  port: 27017    
  bindIp: 127.0.0.1,xx.xx.xx.xx

processManagement:
   fork: true
security:
  authorization: enabled

Output of cat /proc/<PId>/limits:

cat /proc/17694/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             112088               112088               processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       112088               112088               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

ps xua | grep java results :::

  root     17142  0.4  3.1 5645648 453180 pts/0  Sl   Sep04   2:27 java -d64 -Xms1024m -Xmx2048m -server -cp bin:lib/* javaclassname -resourceDir /xxx/bin -dataDir /xxx/data
   root     17491 22.1 11.2 5653432 1610716 pts/0 Sl   Sep04 115:14 java -d64 -Xms1024m -Xmx2048m -server -cp bin:lib_new/* javaclassname -resourceDir /xxx/bin -dataDir /xxx/data -outConfigPath /xxx/xxx.xml -outCatBrandPath /xxx/xxx.xml -outCatAndSunCatPath /xxx/xxx.xml
   xxx      19223  3.5 22.1 20396560 3184668 ?    Sl   Aug28 398:11 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/elasticsearch-5.4.0 -cp /elasticsearch-5.4.0/lib/* org.elasticsearch.bootstrap.Elasticsearch
   root     21617  2.2 21.7 18549248 3124416 ?    Sl   Aug28 247:34 /usr/bin/java -Djava.util.logging.config.file=/opt/apache-tomcat-8.0.30/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/apache-tomcat-8.0.30/endorsed -classpath /opt/apache-tomcat-8.0.30/bin/bootstrap.jar:/opt/apache-tomcat-8.0.30/bin/tomcat-juli.jar -Dcatalina.base=/opt/apache-tomcat-8.0.30 -Dcatalina.home=/opt/apache-tomcat-8.0.30 -Djava.io.tmpdir=/opt/apache-tomcat-8.0.30/temp org.apache.catalina.startup.Bootstrap start

swapon -s output :

Filename                                Type            Size    Used    Priority
12
  • It seems that your server is running out of memory and the kernel out-of-memory killer (OOM-killer) decides to terminate the mongodb process to safeguard the operating system. Please include some indicators like memory usage just before and just after launching your java application, output of free -m and/or top, and java memory sizing (-Xms, Xmx, etc.) – André Fernandes Sep 4 '17 at 16:51
  • Also provide all the configuration from mongod.conf – John Mahowald Sep 4 '17 at 17:00
  • top command results are as follows : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17694 root 20 0 372624 182312 6484 S 24.1 1.3 0:02.12 mongod 17491 root 20 0 5320268 384832 20788 S 6.0 2.7 0:16.05 java 1 root 20 0 33472 1360 60 S 0.0 0.0 0:39.40 init 2 root 20 0 0 0 0 S 0.0 0.0 0:01.01 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:14.28 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H – Sri Sep 4 '17 at 17:09
  • Hi John, I have added the mongod.conf to the question. – Sri Sep 4 '17 at 17:19
  • What is the output of cat /proc/PID/limits where PID is the process id of mongo? Could be a ulimit issue. – Jason Martin Sep 4 '17 at 17:27
0

It seems that the OS is running out of physical memory, and since your server doesn't seem to have any swap configured, the kernel's out-of-memory killer (OOM-killer) kicks in and terminates processes to free up some RAM (in this case it's mongodb, but could be any other).

You should create a swap partition on that server.

4
  • Hi Andre, Thanks for your suggestion. I created a swap partition but system goes down. Is it because of this swap partition? – Sri Sep 7 '17 at 5:22
  • Hi Andre, My server is of 14GB. I created a swap partition of 4GB, Does this makes sense? pls advice... – Sri Sep 7 '17 at 5:30
  • Check if your swap partition is actually in use with swapon -s, it should show something. Check the following guide for instructions on how to configure swap on Linux: thegeekstuff.com/2010/08/how-to-add-swap-space/… . If your swap is properly configured and in use, it's possible that it is too small, check the usage with free -m and if it's becoming full try increasing its size to the same amount as the server's RAM. – André Fernandes Sep 7 '17 at 10:04
  • Thank you so much for your help Andre.. I will try to increase the size as and when required currently OOM-killer is not killing my process. – Sri Sep 7 '17 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.