How to inspect why my service process be killed by Linux?

Question

I have a vital service running a Ubuntu(20.04.1) server. Recently it is always killed by OS.

At first I guess that probably is resulted by the OOM(out of memory) operation of OS, so I modified the systemd service unit file(my_app.service) of my app, and add a option OOMScoreAdjust=-1000. Of course followed by a systemctl daemon-reload.

But my app still is killed by OS!

Now I have to inspect the real reason why OS always kills my app.

BTW, there is 2G RAM and 4G swap space. When my app being killed, almost entire swap space is free. My app should is a good program because it is running normally on another Ubuntu(20.04) server that has 4G RAM and 4G swap.

How to find out the real reason? (/proc/sys/vm/swapness is 65)

Any hints will be appreciated!

Answer 1

I think you can fix things with OOMPolicy not with OOMScoreAdjust please refer this page https://www.freedesktop.org/software/systemd/man/systemd.service.html#

OOMPolicy=
Configure the out-of-memory (OOM) kernel killer policy. Note that the userspace OOM killer systemd-oomd.service(8) is a more flexible solution that aims to prevent out-of-memory situations for the userspace, not just the kernel. On Linux, when memory becomes scarce to the point that the kernel has trouble allocating memory for itself, it might decide to kill a running process in order to free up memory and reduce memory pressure. This setting takes one of continue, stop or kill. If set to continue and a process of the service is killed by the kernel's OOM killer this is logged but the service continues running. If set to stop the event is logged but the service is terminated cleanly by the service manager. If set to kill and one of the service's processes is killed by the OOM killer the kernel is instructed to kill all remaining processes of the service too, by setting the memory.oom.group attribute to 1; also see kernel documentation.

DefaultOOMPolicy=
Configure the default policy for reacting to processes being killed by the Linux Out-Of-Memory (OOM) killer or systemd-oomd. This may be used to pick a global default for the per-unit OOMPolicy= setting. See systemd.service for details. Note that this default is not used for services that have Delegate= turned on.

DefaultOOMScoreAdjust=
Configures the default OOM score adjustments of processes run by the service manager. This defaults to unset (meaning the forked off processes inherit the service manager's OOM score adjustment value), except if the service manager is run for an unprivileged user, in which case this defaults to the service manager's OOM adjustment value plus 100 (this makes service processes slightly more likely to be killed under memory pressure than the manager itself). This may be used to pick a global default for the per-unit OOMScoreAdjust= setting. See systemd.exec for details. Note that this setting has no effect on the OOM score adjustment value of the service manager process itself, it retains the original value set during its invocation.