1

When a process tries to allocate more memory than available, it is usually very hard to handle this situation in a good way. In some cases programs might be able to free non-essential memory (e.g. caches), but it most cases it will lead to fatal exceptions (see also [1]), to swapping and massive system slow down or even to kill (pretty arbitrary) processes by using the OOM killer.

On a single user system all options are bad but at least you can only hurt yourself (after all you are responsible for all code you run on your machine).

On a multi user system however, I'd expect that it might be possible to make the OOM killer kill certain processes possibly owned by other users. There are certain restrictions, because the OOM killer tries to find a "process that is using a large amount of memory but is not that long lived" (see here).

Nevertheless imaging a situation where a process requires x MB of memory and has not lived for very long time (low CPU time) and is owned by user X and x is significantly more than there is free memory on the system, user Y might be able to allocate more memory than is available and cause the system to kill X's process (because it is using more memory).

This situation sounds way more scary than similar implications on a single user machine. One could set memory limitations per user or even use containers to separate processes even more. But as far as I understand only setting the limit to total_memory / number_of_users would solve the problem. But when setting such a limit one looses all benefits from a multi user system. The machine actually is then similar to multiple single user machines in one box. One typically wants to allow one user to use more memory at peak time, because most of the time users will need less memory than average.

I'm mostly interested solving this issue in situations with large calculations with huge amounts of data. I guess in case of web servers one might be able to estimate better how much memory is need as there are many small operations and not few large ones. But even in this case, I've heard that in normal situations only 50% of your memory should be filled to avoid out of memory issues during peaks. Isn't this a waste of 50% of your memory?

In particular I am thinking to host a Jupyter hub or something similar. But I don't want the users to kill each others processes.

Is there any other solution to mitigate this problem? How do huge cloud providers like Travis CI deal with this kind of problems?

3 Answers 3

3

I believe nowadays the recommended way to limit the impact of one task on others is use a cgroup to limit the memory it can use. The easiest way I found is to use systemd-run. For example, this starts ls with a ridiculously low memory limit of 1000 bytes:

sudo systemd-run --uid $USER --gid $UID -p MemoryMax=1000 ls

Running journalctl -f, you will see that process has been killed immediately:

Memory cgroup out of memory: Kill process 3543 ((ls)) score 2503000 or sacrifice child
Killed process 3543 ((ls)) total-vm:205348kB, anon-rss:5612kB, file-rss:4276kB, shmem-rss:0kB

Then of course you will want to oversubscribe the machine, maybe giving each task 70% of available memory? This will allows peaks in memory consumption, while still quickly killing processes that use an unreasonable amount of memory.

2

Disable the oom killer (vm.oom-kill = 0 in a file read by sysctl) and disable memory overcommit (vm.overcommit_memory = 2). Any process that tries to allocate more memory than available will be killed. No other process will be killed. All memory can be used; no memory is wasted.

This doesn't prevent memory hogs. The memory cgroup has the memory.soft_limit_in_bytes setting which looks like it might be useful (in conjunction with memory.limit_in_bytes and memory.memsw.limit_in_bytes) but I found no detailed explanation of how it operates.

Edit: I looked at how memory.soft_limit_in_bytes is used in the kernel source. If memory cannot be reclaimed nothing happens.

1
  • Maybe this is just a theoretical issue and it works well in practice, but the problem here is that some random process that allocates memory at the wrong time may be killed.
    – Jakob
    Jan 12, 2019 at 15:01
1

You can set the sysctl vm.oom_kill_allocating_task. When this is set, and a process memory request would lead to running out of memory, Linux will kill that process.

From the documentation:

This enables or disables killing the OOM-triggering task in out-of-memory situations.

If this is set to zero, the OOM killer will scan through the entire tasklist and select a task based on heuristics to kill. This normally selects a rogue memory-hogging task that frees up a large amount of memory when killed.

If this is set to non-zero, the OOM killer simply kills the task that triggered the out-of-memory condition. This avoids the expensive tasklist scan.

If panic_on_oom is selected, it takes precedence over whatever value is used in oom_kill_allocating_task.

The default value is 0.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .