Understanding ulimits / process limits, or maybe something else. New processes stop opening (fork errors)

Question

I'm a little bit lost and need some help understand what exactly is happening with my server.

So this is a Proxmox (Debian) server with several LXC containers running in it, and from time to time everything just starts failing because it seems that new processes/childrens are unable to open. The syslog starts being filled with messages like this:

May 24 18:19:44 pvirtual08 ksmtuned[1645]: /usr/sbin/ksmtuned: fork: retry: Resource temporarily unavailable
May 24 18:19:46 pvirtual08 pve-firewall[4013]: status update error: command 'iptables-save' failed: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 449.
May 24 18:19:47 pvirtual08 pvestatd[4012]: command 'lxc-info -n 124 -p' failed: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 449.
May 24 18:19:47 pvirtual08 pvestatd[4012]: command 'lxc-info -n 404 -p' failed: open3: fork failed: Resource temporarily unavailable at /usr/share/perl5/PVE/Tools.pm line 449.

until eventually the entire server just crashes. It seems clear that the server stops being able to open new processes after some time but I don't quite understand why. Everything I read points to either a ulimit being reached, or server being out of RAM. Last time I checked the server was no where near having it's RAM full. Regarding ulimits, and this is where I'm a bit lots, from what I can tell that is not being reached either.

This is the current values for ulimit -a

root@pvirtual08:/var/log# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514673
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 514656
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Last time it happened I checked the number of processes running with

ps -eLf | grep -v root | wc -l

and it was no where near the limit either, but maybe I'm just counting wrong, or checking the wrong limit.

Is there a way I can know which limit is being reached exactly for the server to stop opening new forks? I way to monitor the current usage vs limit so I can do a monitoring script for example?

I apologize for any dumb questions but this is a bit new and confusing to me, the way the ulimits work.