I'm a little bit lost and need some help understand what exactly is happening with my server.
So this is a Proxmox (Debian) server with several LXC containers running in it, and from time to time everything just starts failing because it seems that new processes/childrens are unable to open. The syslog starts being filled with messages like this:
May 24 18:19:44 pvirtual08 ksmtuned: /usr/sbin/ksmtuned: fork: retry: Resource temporarily unavailable May 24 18:19:46 pvirtual08 pve-firewall: status update error: command 'iptables-save' failed: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 449. May 24 18:19:47 pvirtual08 pvestatd: command 'lxc-info -n 124 -p' failed: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 449. May 24 18:19:47 pvirtual08 pvestatd: command 'lxc-info -n 404 -p' failed: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 449.
until eventually the entire server just crashes. It seems clear that the server stops being able to open new processes after some time but I don't quite understand why. Everything I read points to either a ulimit being reached, or server being out of RAM. Last time I checked the server was no where near having it's RAM full. Regarding ulimits, and this is where I'm a bit lots, from what I can tell that is not being reached either.
This is the current values for ulimit -a
root@pvirtual08:/var/log# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514673 max locked memory (kbytes, -l) 65536 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514656 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Last time it happened I checked the number of processes running with
ps -eLf | grep -v root | wc -l
and it was no where near the limit either, but maybe I'm just counting wrong, or checking the wrong limit.
Is there a way I can know which limit is being reached exactly for the server to stop opening new forks? I way to monitor the current usage vs limit so I can do a monitoring script for example?
I apologize for any dumb questions but this is a bit new and confusing to me, the way the ulimits work.