Questions tagged [oom]
Linux Out-of-memory Killer
109
questions
67
votes
3
answers
8k
views
Rsync triggered Linux OOM killer on a single 50 GB file
I have a single 50 GB file on server_A, and I'm copying it to server_B. I run
server_A$ rsync --partial --progress --inplace --append-verify 50GB_file root@server_B:50GB_file
Server_B has 32 GB ...
- 833
46
votes
7
answers
39k
views
How do I prevent Linux from freezing when out of memory?
Today I (accidentally) ran some program on my Linux box that quickly used a lot of memory. My system froze, became unresponsive and thus I was unable to kill the offender.
How can I prevent this in ...
- 561
38
votes
4
answers
32k
views
Turn off the Linux OOM killer by default?
The OOM killer on Linux wreaks havoc with various applications every so often, and it appears that not much is really done on the kernel development side to improve this. Would it not be better, as a ...
- 3,725
31
votes
8
answers
34k
views
How to get the Linux OOM killer to not kill my process?
How do I get the Linux OOM killer to not kill my processes when physical memory is low but there is plenty of swap space?
I have disabled OOM killing and overcommit with sysctl vm.overcommit_memory=2....
- 435
20
votes
4
answers
15k
views
Linux oom situation (32 bit kernel)
I have continous oom&panic situation unresolved. I am not sure system fills up all the ram (36GB). Why this system triggered this oom situation? I suspect it as related to lowmem zone in 32 bit ...
- 698
14
votes
1
answer
21k
views
How is kernel oom score calculated?
Looked on The Google, and couldn't find anything that explained how the score in proc/<pid>/oom_score is calculated. Why use this score instead of just using the total memory used?
- 309
14
votes
2
answers
14k
views
OOM despite available memory (cache)
We've been running into the OOM killer despite nearly half of our memory being used for the FS cache. We've been logging memory stats once per minute (as reported by top), but there seems to be ...
- 1,665
13
votes
4
answers
13k
views
How to use kdump/crash to investigate an OOM issue?
The problem
A server crashed after multiple "Out of memory" messages and I am trying to pinpoint the culprit. If it is in userland - which process. If it is in the kernel - which kernel module.
...
- 8,000
13
votes
2
answers
15k
views
Get a core dump of/debugging a process killed by oom-killer
Is there any way to get a core dump of, or be able to debug a process that has been killed by oom-killer?
Or even set oom-killer to try to kill a process using ABRT instead?
- 131
11
votes
3
answers
25k
views
OOM killer killing things with plenty(?) of free RAM
The OOM killer seems to be killing things despite having more than enough free RAM on my system:
(Full resolution)
(Full resolution)
27 minutes and 408 processes later, the system started ...
- 1,371
10
votes
3
answers
7k
views
Why are applications in a memory-limited LXC container writing large files to disk being killed by the OOM?
EDIT2: This problem appears to exist as well under 3.8.0-25-generic #37-Ubuntu SMP
EDIT: I modified the question from the original title of "Why would the Linux Out of Memory manager be triggered by ...
- 1,136
10
votes
2
answers
6k
views
Baffling Memory leak. What is using ~10GB of memory on this system?
After running for about 18 hours, this system is using ~10GB of memory, causing the OOM-killer to be triggered when we run our usual tasks:
# free -h
total used free ...
- 3,225
10
votes
1
answer
9k
views
Linux process killed even though enough memory available
I am investigating why two of our processes were killed by the Linux OOM killer - even though there seems to have been enough RAM and plenty of SWAP available at both times.
When I interpret it as by ...
- 323
9
votes
5
answers
18k
views
How to reliably take Java Heap Dumps?
My team is running into difficulties when trying to take good heap dumps triggered by OutOfMemoryErrors. For specific reasons we are currently taking the dumps with jmap called from a bash script ...
- 153
8
votes
5
answers
4k
views
Prevent Linux's OOM from killing apache on our webserver
We have a debian linux webserver. It's just running apache2. Our mysql server is on another host. However we sometimes run cron tasks on the webserver to do regular tasks.
However recently one of the ...
- 32.3k
8
votes
1
answer
3k
views
Forensic Analysis of the OOM-Killer
Ubuntu's Out-Of-Memory Killer wreaked havoc on my server, quietly assassinating my applications, sendmail, apache and others.
I've managed to learn what the OOM Killer is, and about its "badness" ...
- 274
7
votes
2
answers
33k
views
Enough memory but OOM killer invoked?
It has been a long time since my old friend the OOM killer came to visit me. Now it is coming by once a week and I cannot figure out why it’s triggered. It doesn’t seem that I’m missing memory. What ...
- 93
6
votes
3
answers
3k
views
OOM invoked with plenty of free swap
I am running an ARM router with i2p and tor on it - a Netgear R7000.
Of course I've added a full 512 MB of SWAP to it to prevent OOMs, understanding that it could slow down the system... But then I ...
- 234
6
votes
1
answer
2k
views
How can there ever be an OOM scenario on Linux (heuristics behind OOM killer)?
I am aware of the concept of virtual memory. And with demand paging (depending on vm.overcommit_memory) you are able to allocate more memory than available RAM. Unless you "touch" a page nothing ...
- 173
6
votes
1
answer
887
views
fio 3.23 core dumps when bench-marking many small files
I have been asked to come up fio benchmark results for this test dataset: 1048576x1MiB. So, overall size is 1TiB. The set contains 2^20 1MiB files. The server runs CentOS Linux release 7.8.2003 (Core)...
- 63
6
votes
4
answers
413
views
LInux OOM custom response
I'm running Apache webserver and would like to improve a little bit how the OOM situation is handled.
I'm avare of the OOM scores and already did some customizations in that matter so when something ...
- 315
5
votes
3
answers
18k
views
what do these messages suggest (linux kernel dump, seems OOM)?
I got the following errors from system log (/var/log/messages). what they indicate? And how to solve them?
Sep 30 05:19:35 node23 kernel: Mem-info:
Sep 30 05:19:35 node23 kernel: Node 0 DMA per-cpu:
...
- 949
5
votes
1
answer
5k
views
apache httpd process be killed by kernel oom
some apache httpd process's memory and cpu usage too high, be killed by kernel, but I don't know why httpd use such many memory. I need some method to analyse or some method to reproduct.
some ...
- 51
5
votes
1
answer
1k
views
Disable Linux kernel from killing postgresql process
I have standard 'out-of-the-box' installation of
Linux version 3.0.1.stk64 ([email protected]) (gcc version 4.5.1 20100924 (Red Hat 4.5.1-4) (GCC) ) #1 SMP Sat Aug 13 12:53:46 EDT 2011
It ...
- 151
4
votes
2
answers
4k
views
Linux OOM-killer acting despite plenty available memory
Once a week or so the OOM-killer shoots down a postgres process in my server, despite that 'free' states it has plenty of available memory.
I have read several threads here and there, but cant see ...
4
votes
2
answers
6k
views
What's the technique to find out bottleneck?
I setup a wordpress on Digital Ocean for the smallest droplet for a few months.
Recently, my blog gone viral (1000+ facebook shares) and the server take 50+ seconds to respond.(Google Analytics show ...
- 143
4
votes
1
answer
3k
views
How to find which cgroup caused OOM?
I'm using cgroup to partition my processes and I'm getting Out Of Memory messages in my kernel logs.
However, I can't find which partition causes them. I've checked the memory controller cgroup but ...
- 269
4
votes
4
answers
8k
views
Ubuntu how to detect if OOM killer have run
We use Nagios to monitor our servers. Now.. linux uses all the RAM we give them so there is no way to detect if we are about to run out of memory. We have a couple of times had some server crashes due ...
- 221
4
votes
2
answers
2k
views
OOM killer goes insane
On our cluster we would sometimes have nodes go down when a new process would request too much memory. I was puzzled why the OOM killer does not just kill the guilty process.
The reason turned out to ...
- 2,495
4
votes
2
answers
427
views
Out of memory at 18% usage : where the ram goes?
This is related to : Out of memory at 72% usage
It looks to be the same problem but the question is slightly different : Where my memory goes ? I have 18% memory usage and my OOM Killer is killing ...
- 659
3
votes
4
answers
1k
views
How to run DNF without going OOM on GCP e2-micro
I'm trying to run Fedora Cloud Base 36 on an e2-micro in the US central region of Google Cloud Platform (GCP). I haven't upgraded packages in a couple months so I tried to do that today. When I run:
$ ...
google-cloud-platform
3
votes
1
answer
2k
views
Adjust oom_score at process launch
I've got a few servers that have begun oom-killing their backup processes and, while I understand that encountering the oom condition is quite bad in itself, I need this process to not die so that ...
- 2,151
3
votes
2
answers
9k
views
Long page allocation stalls on Linux – why does this happen?
I have a problem (which I can reliably reproduce) on a bunch of Linux hosts, where the system becomes completely unresponsive after a process aggressively consumes memory. I see things like this in ...
- 304
3
votes
2
answers
1k
views
GKE's Container-Optimized OS Out-Of-Memory freezes
I have issue with Container-Optimized OS on GKE. If I run this simple command https://pastebin.com/raw/0WPAnAzn to consume all the RAM, at some point host freezes and doesn't respond to anything. ...
3
votes
3
answers
974
views
MySQL - Identifying a never ending, server crashing query
My server keeps crashing from OOM errors cause by MySQL. I am convinced there is a query on this site that is wrong, too long and causes this issue. My server and MySQL logs show very little strain ...
- 231
3
votes
4
answers
7k
views
mysqld crashing almost every hour - mysqld invoked oom-killer
I am running wordpress on a centOS digital ocean droplet
my mysql keeps crashing almost every hour -- with an out of memory error
manually restarting mysqld brings it back up via sudo service mysqld ...
- 31
3
votes
1
answer
2k
views
How to inspect why my service process be killed by Linux?
I have a vital service running a Ubuntu(20.04.1) server. Recently it is always killed by OS.
At first I guess that probably is resulted by the OOM(out of memory) operation of OS, so I modified the ...
- 179
3
votes
0
answers
591
views
Running out of memory with plenty of memory and swap
I have a Debian-based system that is running out of memory but there appears to be plenty of free memory. The box will run for about 6-12 days and then it will begin killing anything allocating memory ...
- 31
3
votes
1
answer
7k
views
Linux OOM: Trying to understand memory statistics / mismatch with expected values
I am trying to understand why the OOM killer has reaped a process on a KVM virtual machine running with 1 gigabyte of memory. While I do already know that running Linux machines without swap is not ...
- 33
3
votes
2
answers
4k
views
Slab reclaimable memory is not freed when needed
Correct me if I am wrong, but to my understanding slab reclaimable holds cached kernel objects which can be freed if needed. So if application needs to allocate more space, even if the 'free' memory ...
- 161
3
votes
0
answers
836
views
Why can't I malloc as much memory as `free` reports as being free?
The following C program allocates slices of 10MB of memory until it is killed by the Out of Memory killer.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <...
- 191
2
votes
2
answers
2k
views
NFS server with out of memory
I have multiple machines sharing home directory via NFS share used by
6-10 users. All machines are used to run computational experiments including
the one with NFS server. Although it is very rare but ...
- 29
2
votes
1
answer
8k
views
An oom killer that I cannot explain
I am not able to understand why would kernel issue this oom killer when I see enough memory is available:
I say enough memory is available after looking at
Normal
DMA
Normal free lines
This is an ...
- 457
2
votes
3
answers
2k
views
AWS WordPress site - OOM kills Apache
This question was originally asked on StackOverflow.com, and I've duplicated it to the more appropriate ServerFault.com site. The original question, which I've voted to close, can be found here
I ...
- 123
2
votes
3
answers
2k
views
OOM kills a process while there is still a lot of available memory
I am aware that there are many threads like this one, but none of them actually give a methodology on how to read an OOM message.
I have a db that is being monitored through NRPE for available ...
- 273
2
votes
1
answer
4k
views
Using linux oom_score_adj to aim the oom daemon at the right target
On Ubuntu 11.04, /proc/[pid] contains oom_score_adj, but man proc describes the old oom_adj.
What I need to arrange is that processes started by 'pbs' or 'gridengine' are killed before anything else ...
- 2,403
2
votes
1
answer
259
views
Weird console problem in linux, usually right before OOM
So I've noticed this happen more than once.
If I remember correctly, this usually happens right before OOM, and/or kernel panic that if you type into a tty and then try to backspace it, the same ...
- 21
2
votes
1
answer
10k
views
OOM killer invoked despite tons of memory available? [duplicate]
I have a server where the OOM killer has been invoked once or twice almost each night since two weeks ago. The server should have more than enough memory it could free (cache/buffers, inactive_file ...
- 121
2
votes
2
answers
3k
views
Why is OOM killer hitting here
We have some VMs running kernel 2.26.26-2 under xen 4.0.3 with different CPU and RAM amounts and on some seeing almost random OOM events each rendering the VM unusable. Output is as follows, I have ...
- 1,217
2
votes
2
answers
2k
views
lighttpd eating all memory and CPU
So my lighttpd was running fine for more than a year. Two days ago I decided to install Wordpress 3.0.1 (PHP was already in use by other tools).
So after some hours of very few hits (i might get 10 ...
- 4,119