I had problem with a process spawning very many threads due to a bug, eating up memory, causing heavy swapping to the swap partition. Thus, I switched off swap – which is recommended in server clusters anyway – to let the buggy program fail early. To my surprise, this didn't improve things at all! Similarly, setting
vm.swappiness to 0 didn't help.
iotop revealed heavy reading from disk (almost no writing).
I suspect that the Linux kernel is still swapping code pages in memory, and re-reads overwritten code pages from disk when they are needed. When RAM is very low, this happens very frequently, causing thrashing, making the computer almost unresponsive.
How can one prevent this unfortunate and potentially dangerous situation? I tried switching off swap, cgroups for memory and CPU constraints for that process, and ulimit for max. 30 processes. Nothing even improved the situation. (Well, this is not entirely true: a RAM limit which left hundreds of MB permanently unused did help.)
In particular: Can I prevent swapping code segments in memory, and let the RAM-requesting process simply get a NULL for their next malloc?