1

I was testing how an Ubuntu 16.04 machine handles low memory conditions and had several suspended jobs that that used up almost all the RAM on the server. I was still able to run normal bash commands when I left work last night. In the morning (without anyone doing anything to the server) all commands (ps, free, ls, etc.) gave the following error:

-bash: fork: Cannot allocate memory

I was eventually able to run jobs and kill %1 to recover the machine. It was lucky that I kept an SSH session open, because further SSH connections failed (the client simply didn't show a command prompt after authentication).

The fact that regular userland programs can effectively bring down the entire server to the point where, if there wasn't an existing SSH session, it would have to be hard-rebooted, is highly concerning! My understanding was that Linux (or any OS, really) should kill processes to free up RAM before the entire system gets to such an unusable state. How do I make it do that?

  • Can you post the output of free? – Khaled Feb 21 '18 at 9:35
  • did you reconfigure the sysctl parameters vm.overcommit_memory and/or vm.overcommit_ratio? Seeing "cannot allocate memory" is somewhat unusual on a linux system in standard configuration. – Andreas Rogge Feb 21 '18 at 9:45
  • @Khaled free failed with -bash: fork: Cannot allocate memory – EM0 Feb 21 '18 at 13:07
  • @AndreasRogge No, I didn't change them. This VM is based on AWS standard image. I just checked sysctl and I have vm.overcommit_memory = 0, vm.overcommit_ratio = 50. – EM0 Feb 21 '18 at 13:08
  • I now got the server into the same bad state again by using almost all available RAM. jobs shows nothing now and kill -9 %1' returns -bash: kill: %1: no such job` (though there should be jobs running). I also get -bash: wait_for: No record of process 10728 in response to some jobs and kill commands. – EM0 Feb 21 '18 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.