2

I'm running a CentOS 6.5 64 bit OS with Tomcat and one web application. I use Apache (httpd) as a reverse proxy (ProxyPass) to the application. Until recently, it (Tomcat) has been running fine without crashing. Recently, I added a Joomla site and PHP (obviously) to the http server. It was shortly after that where I started getting Tomcat crashes.

When it crashes, Tomcat doesn't leave evidence in the Catalina.out. I finally found output in /var/log/messages. I was hoping someone could have a cursory look and point me in the right direction:

============

Jul 23 01:19:31 myservername kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 
Jul 23 01:19:31 myservername kernel: httpd cpuset=/ mems_allowed=0 
Jul 23 01:19:31 myservername kernel: Pid: 6104, comm: httpd Not tainted
2.6.32-431.1.2.0.1.el6.x86_64 #1

Jul 23 01:19:31 myservername kernel: Call Trace:
Jul 23 01:19:31 myservername kernel: [<ffffffff810d05c1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Jul 23 01:19:31 myservername kernel: [<ffffffff81122970>] ? dump_header+0x90/0x1b0
Jul 23 01:19:31 myservername kernel: [<ffffffff8122799c>] ? security_real_capable_noaudit+0x3c/0x70
Jul 23 01:19:31 myservername kernel: [<ffffffff81122df2>] ? oom_kill_process+0x82/0x2a0
Jul 23 01:19:31 myservername kernel: [<ffffffff81122d31>] ? select_bad_process+0xe1/0x120
Jul 23 01:19:31 myservername kernel: [<ffffffff81123230>] ? out_of_memory+0x220/0x3c0
Jul 23 01:19:31 myservername kernel: [<ffffffff8112fb4c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Jul 23 01:19:31 myservername kernel: [<ffffffff81167aaa>] ? alloc_pages_current+0xaa/0x110
Jul 23 01:19:31 myservername kernel: [<ffffffff8111fd67>] ? __page_cache_alloc+0x87/0x90
Jul 23 01:19:31 myservername kernel: [<ffffffff8111f74e>] ? find_get_page+0x1e/0xa0
Jul 23 01:19:31 myservername kernel: [<ffffffff81120d07>] ? filemap_fault+0x1a7/0x500
Jul 23 01:19:31 myservername kernel: [<ffffffff8114a094>] ? __do_fault+0x54/0x530
Jul 23 01:19:31 myservername kernel: [<ffffffff8114a667>] ? handle_pte_fault+0xf7/0xb00
Jul 23 01:19:31 myservername kernel: [<ffffffff811214a0>] ? generic_file_aio_read+0x380/0x700
Jul 23 01:19:31 myservername kernel: [<ffffffff8114b29a>] ? handle_mm_fault+0x22a/0x300
Jul 23 01:19:31 myservername kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
Jul 23 01:19:31 myservername kernel: [<ffffffff8100988e>] ? __switch_to+0x26e/0x320
Jul 23 01:19:31 myservername kernel: [<ffffffff81527920>] ? thread_return+0x4e/0x76e
Jul 23 01:19:31 myservername kernel: [<ffffffff8152d46e>] ? do_page_fault+0x3e/0xa0
Jul 23 01:19:31 myservername kernel: [<ffffffff8152a825>] ? page_fault+0x25/0x30
Jul 23 01:19:31 myservername kernel: Mem-Info:
Jul 23 01:19:31 myservername kernel: Node 0 DMA per-cpu:
Jul 23 01:19:31 myservername kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jul 23 01:19:31 myservername kernel: Node 0 DMA32 per-cpu:
Jul 23 01:19:31 myservername kernel: CPU    0: hi:  186, btch:  31 usd:  30
Jul 23 01:19:31 myservername kernel: active_anon:112069 inactive_anon:56 isolated_anon:0
Jul 23 01:19:31 myservername kernel: active_file:17 inactive_file:465 isolated_file:0
Jul 23 01:19:31 myservername kernel: unevictable:0 dirty:1 writeback:0 unstable:0
Jul 23 01:19:31 myservername kernel: free:1181 slab_reclaimable:1556 slab_unreclaimable:5516
Jul 23 01:19:31 myservername kernel: mapped:78 shmem:58 pagetables:2575 bounce:0
Jul 23 01:19:31 myservername kernel: Node 0 DMA free:2040kB min:84kB low:104kB high:124kB active_anon:13500kB inactive_anon:0kB active_file:12kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:0kB dirty:0kB writeback:0kB mapped:24kB shmem:0kB slab_reclaimable:40kB slab_unreclaimable:76kB kernel_stack:16kB pagetables:48kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no
Jul 23 01:19:31 myservername kernel: lowmem_reserve[]: 0 489 489 489
Jul 23 01:19:31 myservername kernel: Node 0 DMA32 free:2684kB min:2784kB low:3480kB high:4176kB active_anon:434776kB inactive_anon:224kB active_file:56kB inactive_file:1848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500948kB mlocked:0kB dirty:4kB writeback:0kB mapped:288kB shmem:232kB slab_reclaimable:6184kB slab_unreclaimable:21988kB kernel_stack:944kB pagetables:10252kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:130 all_unreclaimable? yes
Jul 23 01:19:31 myservername kernel: lowmem_reserve[]: 0 0 0 0
Jul 23 01:19:31 myservername kernel: Node 0 DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2040kB
Jul 23 01:19:31 myservername kernel: Node 0 DMA32: 193*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2684kB
Jul 23 01:19:31 myservername kernel: 547 total pagecache pages
Jul 23 01:19:31 myservername kernel: 0 pages in swap cache
Jul 23 01:19:31 myservername kernel: Swap cache stats: add 0, delete 0, find 0/0
Jul 23 01:19:31 myservername kernel: Free swap  = 0kB
Jul 23 01:19:31 myservername kernel: Total swap = 0kB
Jul 23 01:19:31 myservername kernel: 131068 pages RAM
Jul 23 01:19:31 myservername kernel: 5503 pages reserved
Jul 23 01:19:31 myservername kernel: 13454 pages shared
Jul 23 01:19:31 myservername kernel: 119917 pages non-shared
Jul 23 01:19:31 myservername kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Jul 23 01:19:31 myservername kernel: [  325]     0   325     2727      168   0     -17         -1000 udevd
Jul 23 01:19:31 myservername kernel: [  602]     0   602     2726      167   0     -17         -1000 udevd
Jul 23 01:19:31 myservername kernel: [  878]     0   878    23300       78   0     -17         -1000 auditd
Jul 23 01:19:31 myservername kernel: [  894]     0   894    62368      729   0       0             0 rsyslogd
Jul 23 01:19:31 myservername kernel: [ 1027]     0  1027    29325      153   0       0             0 crond
Jul 23 01:19:31 myservername kernel: [ 1050]     0  1050     1016       21   0       0             0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1052]     0  1052     1016       22   0       0             0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1054]     0  1054     1016       22   0       0             0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1056]     0  1056     1016       22   0       0             0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1058]     0  1058     1016       21   0       0             0 mingetty
Jul 23 01:19:31 myservername kernel: [ 1189]    38  1189     7679      158   0       0             0 ntpd
Jul 23 01:19:31 myservername kernel: [19331]    99 19331    11317      405   0       0             0 openvpn
Jul 23 01:19:31 myservername kernel: [19336]     0 19336     2726      167   0     -17         -1000 udevd
Jul 23 01:19:31 myservername kernel: [27296]     0 27296    16652      178   0     -17         -1000 sshd
Jul 23 01:19:31 myservername kernel: [29754]     0 29754     1016       22   0       0             0 mingetty
Jul 23 01:19:31 myservername kernel: [14500]     0 14500    20321      226   0       0             0 master
Jul 23 01:19:31 myservername kernel: [14503]    89 14503    20384      233   0       0             0 qmgr
Jul 23 01:19:31 myservername kernel: [ 3155]     0  3155    27042       66   0       0             0 mysqld_safe
Jul 23 01:19:31 myservername kernel: [ 3257]    27  3257   160134     4405   0       0             0 mysqld
Jul 23 01:19:31 myservername kernel: [ 6102]     0  6102    76169     1432   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6104]    48  6104   104399     5480   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6105]    48  6105   104565     5583   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6106]    48  6106   104301     5322   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6107]    48  6107   104630     5682   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6108]    48  6108   104491     5570   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6109]    48  6109   104563     5570   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6110]    48  6110   104546     5639   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6111]    48  6111   104362     5469   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6171]    48  6171   101911     3019   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6186]    48  6186   101530     2632   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6522]    48  6522   104451     5479   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 6835]   501  6835   317935    55623   0       0             0 java
Jul 23 01:19:31 myservername kernel: [ 7422]    48  7422   104365     5444   0       0             0 httpd
Jul 23 01:19:31 myservername kernel: [ 7656]    89  7656    20341      220   0       0             0 pickup
Jul 23 01:19:31 myservername kernel: Out of memory: Kill process 6835 (java) score 444 or sacrifice child
Jul 23 01:19:31 myservername kernel: Killed process 6835, UID 501, (java) total-vm:1271740kB, anon-rss:222432kB, file-rss:60kB

============

I already raised the heap settings from the default, but that didn't help:

-Xms256m -Xmx512m

I assume that the right direction is to start profiling/monitoring the JVM with one of the tools that comes with the JDK/JRE and see where the leak is. It's just odd that I never had this problem before fiddling with the new Joomla site. I'm not familiar with profiling/monitoring, so I'd appreciate suggestions on which tool to use for my particular situation, given the log file snippet included.

Improve this question

2 Answers 2

Reset to default
5

It seems that there is no more memory available to use on your system. Please check if you have swap memory on your system with below command

free -m

output should be as below if you have swap enabled.

             total       used       free     shared    buffers     cached
Mem:          1876       1798         77          0        119       1440
-/+ buffers/cache:        237       1638
Swap:         4607          0       4607

If you can't see swap in your file, you have to create swapfile. Here is a guide to create and enable swapfile.

http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

Also you can configure javamelody with your tomcat application to identify issues with your tomcat application

--- UPDATE

I've wrote a post about swap memory, if you any question please comment on the below post

https://thinkersbase.blogspot.in/2018/03/create-linux-swap.html

Improve this answer
3
  • 1
    There was no swap configured...good assessment! This is a DigitalOcean 512MB RAM VPS and, as I found out as a result of your assessment, they don't configure swap space on their machines by default. I configured it (digitalocean.com/community/tutorials/…) and rebooted and I don't expect for the problem to come up again...
    – Kevin
    Jul 26, 2014 at 3:59
  • I have swap memory enabled still getting the same error.
    – Manoj
    Jan 28, 2019 at 7:23
  • Please increasing swappiness if set to very low and try again, also provide the output of swapon -s and free -m
    – Vaibhav Panmand
    Jan 28, 2019 at 7:30
2

Here is the cause:

Jul 23 01:19:31 myservername kernel: [<ffffffff81122df2>] ? oom_kill_process+0x82/0x2a0

There is not enough ram. oom_kill_process is the kernel function which kills some processes if there is no ram enough. Something seems leaking, or you have very few memory. Is your swap really enabled and working? (cat /proc/swaps).

Extension: some are saying, that enabling swap makes your system slower. That is false, don't believe that, see my comment.

Improve this answer
4
  • enabling swap will have an effect on the performance, a recommendation is to increase the memory or kill some processes to free up memory. Also, -Xms256m -Xmx512m should be -Xms512m -Xmx512m (note that they are the same) as this might help in seeing the OOM in action earlier than later. Also, note that even though the heap is 512m, the jvm process would also be utilizing memory as well and that might increase based on the workflow, load etc.
    – Khanna111
    Jul 23, 2014 at 14:54
  • @Khanna111GauravKhanna He had a kernel message, not a jvm error. Your (obviously, perfect) jvm arguments are only for the jvm problems, and not for the system memory bottleneck. Your statement about swap is not true, if there is a memory bottleneck on your system, enabling swap makes it faster and much more stabile. Bad performance isn't caused by the swap, swap tries to handle that, which is not always possible.
    – peterh
    Sep 24, 2014 at 9:15
  • I see the stmt of increasing mem led to an implication of increasing mem of tomcat. I meant that we increase the memory of the system. About swap, there are other reasons for not enabling swap such as getting the errors on OOM earlier than later. Enabling swap on a system with memory issues would help no doubt but at the cost of responsiveness if there is too much swapping going on where multiple programs need more RAM to complete. But the reverse is true in cases where the system is idler. Benchmarking is the way to go. Revising to say that enabling swap can have a + or - effect.
    – Khanna111
    Sep 24, 2014 at 18:47
  • For more discussion on swap: please see unix.stackexchange.com/questions/2658/…
    – Khanna111
    Sep 24, 2014 at 18:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .