Questions tagged [oom-killer]

oom-killer Out-Of-Memory killer in the Linux kernel

Filter by
Sorted by
Tagged with
0 votes
0 answers
187 views

Implications of CPU limit on a docker run Java (tomcat) application?

I have had an ongoing problem running a Tomcat java web application in a docker container (which I refer to as a 'task' in this post) hosted in an ECS (elastic container service) on AWS. We notice ...
user avatar
  • 11
0 votes
1 answer
66 views

Out of Memory Mismatch compared to what system rerports

Oct 25 07:41:32 KVM-BOX kernel: memory: usage 255216608kB, limit 255216640kB, failcnt 28058 [root@KVM-BOX ~]# free -hm total used free shared buff/cache available ...
user avatar
3 votes
0 answers
166 views

Running out of memory with plenty of memory and swap

I have a Debian-based system that is running out of memory but there appears to be plenty of free memory. The box will run for about 6-12 days and then it will begin killing anything allocating memory ...
user avatar
  • 31
0 votes
0 answers
519 views

RHEL 8. linux OOM killer invoked to kill postgres process

Database is going to recovery mode whenever the OOM killer is invoked on the postgres process. /var/log/messages output with respect to the issue oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),...
user avatar
0 votes
1 answer
429 views

How do I track down a mysql memory issue on a server running a single WordPress site?

I'm having a weird issue. A few times a week, a website I'm running on Amazon Lightsail comes to a crashing halt and becomes IO bound (I can't usually use ssh when this happens) and the server runs ...
user avatar
  • 432
3 votes
4 answers
2k views

mysqld crashing almost every hour - mysqld invoked oom-killer

I am running wordpress on a centOS digital ocean droplet my mysql keeps crashing almost every hour -- with an out of memory error manually restarting mysqld brings it back up via sudo service mysqld ...
user avatar
0 votes
0 answers
228 views

Monitoring apache workers that are being terminated by oomkiller (kubernetes)

I have a docker container based on php:7.2-apache-stretch running in kubernetes/rancher. The current k8s memory limit is excessively low, so users get errors and truncated pages when worker processes ...
user avatar
0 votes
1 answer
649 views

Slab SReclaimable memory can not be recycled?

CentOS Linux release 7.2.1511 (Core) Linux version 3.10.0-514.26.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP TueJul 4 15:04:05 UTC ...
user avatar
0 votes
1 answer
529 views

Kill the most memory-hungry process when memory gets low [closed]

Sometimes my Linux desktop, laptop, or server run out of RAM because the experimental software I work with occasionally tries to use more RAM than I have available. When this happens, the system and ...
user avatar
  • 231
0 votes
2 answers
689 views

Kubernetes DaemonSet Pod exceeds memory limit but is not killed

We use kubernetes (specifically: openshift) to run our infrastructure in our team. One of the daemonsets, fluentd, is currently causing a lot of trouble, frequently taking full nodes down with huge ...
user avatar
  • 340
2 votes
0 answers
72 views

What abbreviations in OOM Killer memory statistics report means?

When OOM Killer or kernel reports memory state it uses the next abbreviations Node 0 DMA: 26*4kB (M) 53*8kB (UM) 33*16kB (ME) 23*32kB (UME) 6*64kB (ME) 7*128kB (UME) 1*256kB (M) 2*512kB (ME) 0*...
user avatar
  • 163
1 vote
0 answers
480 views

Linux out of memory killer kills process when there is more than enough memory available

We are running a 64-Bit Ubuntu and our 32GB of physical memory is split into 3 zones (DMA: 16MB, DMA32: 4GB and Normal: 30GB). According to the dmesg logs printed below our system ran out of free ...
user avatar
2 votes
1 answer
5k views

Why total-vm is is different than show in table?

I am running into oom_killer activity and cannot undesrand dmesg output [Fri Jan 11 02:11:48 2019] lowmem_reserve[]: 0 2811 64249 64249 [Fri Jan 11 02:11:48 2019] Node 0 DMA32 free:248704kB min:...
user avatar
8 votes
3 answers
5k views

Please help interpret OOM-Killer

We were testing our java application and it was killed by the OOM Killer, logs below. A couple of points: this is a java web application. It runs with a 190G heap. The server is a 240G total memory ...
user avatar
  • 201
0 votes
1 answer
236 views

What are the specific column headers for the "Mem-Info" dump after "oom-killer"

I'm running a Debian 8 Jessie development server. 16GB mem. I've looked at the other posts about oom-killer, but none are really pertinent to what I'm asking here I'm trying to track down what ...
user avatar
0 votes
0 answers
292 views

Strange numbers in Oom killer log

First, the log: [Wed aug 22 01:11:31 2018] java invoked oom-killer: gfp_mask=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0 [Wed aug 22 01:11:31 2018] java ...
user avatar
  • 123
2 votes
3 answers
652 views

Stop a process (script) from invoking oom killer

I had a very unfortunate situation, where a bash script that contained a subtle error gone wild and took all available memory and then started killing other tasks (production apps!) to get even more. ...
user avatar
  • 123
3 votes
2 answers
3k views

OOM killer shut down MySQL although ram usage seems low

Setup EC2 t2.large instance with Ubuntu 16.04 (2 Cores, 8GB RAM) Apache/2.4.18 (Ubuntu) 5.7.20-0ubuntu0.16.04.1 (Ubuntu) Currently this instance hosts multiple Drupal 6 & 7 sites, around 150 ...
user avatar
7 votes
1 answer
9k views

Is it recommended to set OOMScoreAdjust=1000 in systemd for a replaceable service?

systemd has the OOMScoreAdjust option, which allows to adjust the oom-killer score of the started process. To quote from the systemd documentation: OOMScoreAdjust= Sets the adjustment level ...
user avatar
1 vote
0 answers
593 views

Processes is getting killed randomly on REdHat

We have around java virtual machines with limited memory, and MongoDB. Deployed processes are randomly getting killed/terminated abruptly in time. No matter java process, nodejs or mongodb process. We ...
user avatar
0 votes
1 answer
323 views

Number of httpd processes and prefork.c

Hi I got OOM killer problems which caused: [mpm_prefork:error] [pid 18103] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process I had to restart the server to get it working. In /...
user avatar
0 votes
1 answer
906 views

VmRSS used to only about 25%, yet oom-killer strikes

I have a dedicated MySQL server equipped with 128 GB RAM. MySQL recently gets killed by the oom-killer, although MySQL is configured to use 95 GB in the worst case. In my research I came across this: ...
user avatar
  • 105
0 votes
1 answer
203 views

Kernel killing processes with memory available

We have a CentOS5.8 server with 64Gb RAM running normally around 700Mb RAM usage. When I try to do a tar -czf into about a 22Gb archive, the pagecache climbs up to about 22Gb, then the kernel starts ...
user avatar
  • 3
7 votes
2 answers
11k views

How does Docker deal with OOM killer and memory limits?

I have a docker container which starts a simple java (jgroups-based) application through a bash script. The java process is limited through Xmx by 128m, the container is allowed to use 256m (swap is ...
user avatar
  • 239
3 votes
1 answer
1k views

Adjust oom_score at process launch

I've got a few servers that have begun oom-killing their backup processes and, while I understand that encountering the oom condition is quite bad in itself, I need this process to not die so that ...
user avatar
  • 2,072
2 votes
1 answer
4k views

Oracle invoked oom-killer even when RAM and SWAP is plenty free?

Can you please tell me what cause this Oracle process killed? Seems like plenty of RAM free, and plenty of SWAP free. There followed few other oracle processes killed. The VM has the 16G of vMem and 8 ...
user avatar
  • 23
-2 votes
1 answer
128 views

Is there a way for Apache to control how much memory it reserves?

Apache seems to have no control over how much memory it reserves. In MPM prefork you control the maximum number of processes you want to allow, but whatever number you use there, there is always a ...
user avatar
  • 115
1 vote
1 answer
706 views

OOM score adjustements

On Ubuntu 12.04/Java 1.7 I have been reading/searching about OOM killer and generally understood what "proc" man page says especially oom_adj part. It all started because my important "MQ listeners" ...
user avatar
67 votes
3 answers
7k views

Rsync triggered Linux OOM killer on a single 50 GB file

I have a single 50 GB file on server_A, and I'm copying it to server_B. I run server_A$ rsync --partial --progress --inplace --append-verify 50GB_file root@server_B:50GB_file Server_B has 32 GB ...
user avatar
  • 833
0 votes
0 answers
508 views

OOM Killer not triggering?

I'm working in an environment which has just been upgraded to Amazon Linux 2015.03 from Debian 6, quite a big jump. On the new platform a server instance has locked up a few times due to running out ...
user avatar
  • 300
2 votes
1 answer
14k views

Understanding OOM killer logs

I run some processes inside docker container and I use memory limitation for this container. Sometimes some processes inside docker container got killed by OOM killer. I see that in syslog file: ...
user avatar
  • 223
10 votes
2 answers
3k views

View full program command line arguments in OOM killer logs

Is it possible to view full program command line arguments in OOM killler logs ? What I see now in /var/log/syslog is Memory cgroup out of memory: Kill process 29187 (beam.smp) score 998 or sacrifice ...
user avatar
  • 223
5 votes
3 answers
7k views

Mysql crashing, oom-killer, out of memory, tuning issues?

I just moved all of my websites to a new server with 4GB RAM. Almost immediately, mysql started crashing, and at one point, didn't reboot which caused a major outage (since I didn't notice until ...
user avatar
7 votes
1 answer
3k views

linux OOM-kill why?

. Greetings, fellow admins. I post in the hope of shedding light on the OOM-kills that haunt one of my company's machines. I cannot decide if they are legit OOMs or not. It's a Centos 6.x with kernel ...
user avatar
  • 71
17 votes
4 answers
14k views

Linux oom situation (32 bit kernel)

I have continous oom&panic situation unresolved. I am not sure system fills up all the ram (36GB). Why this system triggered this oom situation? I suspect it as related to lowmem zone in 32 bit ...
user avatar
  • 668
1 vote
1 answer
128 views

On SmartOS how to tell if process was killed for OOM

A daemon process of mine gets killed almost immediately, as soon as it starts receiving requests. The application log has absolutely no error output, even when I try to run it in foreground. I have ...
user avatar
  • 113
7 votes
1 answer
10k views

MongoDB getting OOM killed

we are running a mongodb replicaset on three machines. All three machines have around 16GB but only 255MB Swap. Swappiness is left on it's default value 60. The machines are running CentOS 6.4. The ...
user avatar
4 votes
1 answer
2k views

Troubleshooting oom-killer using atop: Is it fixed or not?

I have a Debian system with 4GB of RAM, 2GB of swap and 2 CPUs. I've been experiencing a number of system failures involving oom-killer. The system will be working fine one minute, and the next will ...
user avatar
  • 115
16 votes
1 answer
14k views

Scaling Logstash (with redis/elasticsearch)

Over a cluster of over 12 centos 5.8 servers, I deployed logstash using the native logstash shipper, which sends /var/log/*/*.log back to a central logstash server. We tried using rsyslogd as the ...
user avatar
  • 27.4k