On our server we have different users that need sometimes a big amount of memory (up to 40Gb). The total of our server is 128 Gb, but recently it crashed due to different users taking up together all memory. I couldn't even log in as a sysadmin, server was unresponsive.
For the moment, the only "solution" we found was limiting the amount of memory for each process. Alas it is then still possible to use up the complete 128 Gb, which will lead to another server crash.
Ideally, we want to reserve the last few Gb of the memory for the sysadmin, so that we can always get back on the server and kill some processes manually. In our department huge datasets are used, so we can't restrict too much.
Any idea how we can proceed on this?