Why configure cluster nodes to reboot when out of memory?

Question

I have access to a research HPC cluster which is configured so that if your job tries to use more memory than the node has available the node crashes and automatically reboots. This appears to be common practice, e.g. see https://www.osc.edu/documentation/knowledge_base/out_of_memory_oom_or_excessive_memory_usage

Why would it be configured like this, rather than being configured to just terminate the process(es) requiring too much memory? In both cases you lose the job, but the latter would seem to be better for the cluster as a whole as the node is re-usable faster. Or is it not possible for the OS to guarantee recovering the memory in that case?

Answer 1

Rebooting the node ensures that the node is working properly before the next job is assigned to it. Also, when you run a node out of memory and start swapping it will slow down and may become unresponsive. In this case, they may be using something like IPMI to power cycle the node.

Answer 2

Based on the environment that you are describing and some information that can be found by digging through the link you provided it sounds like the cluster you are using is provisioned as a diskless or stateless system, which means that the entire operating system is loaded into memory from an OS image that is stored remotely.

Ensuring that parallel jobs are killed correctly across several nodes can be a complicated process and ensuring that the killing and cleanup were done correctly can often take more time than simply rebooting a node. Getting a node into a clean state before starting a job is necessary to ensure the highest performance of the cluster.

The exact reasons for configuring a node in this fashion would depend on the provisioning and resource management systems employed on the cluster as well