My MongoDB database was running into problems under load, with the following errors spamming the logs:
[initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable [initandlisten] can't create new thread, closing connection
I've come to the conclusion that I need to raise the "ulimit -u" or "Max processes" setting which were at 1024, and the usage could have been exceeding that given the web frontends launched (not sure how to check this).
I edited /etc/security/limits.conf to add the last two lines (the first two were already there):
* soft nofile 350000 * hard nofile 350000 * soft nproc 30000 * hard nproc 30000
Then I rebooted the system (BTW should I have done that, or should a mongod service restart be enough?)
After reboot, reviewing the process limits for mongod process it seems the soft limit has been ignored:
$ cat /proc/2207/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 1024 30000 processes Max open files 350000 350000 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 273757 273757 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us $ whoami mongod $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 273757 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 350000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
I expected that "Max processes" both hard and soft limits will be at 30000 as per the /etc/security/limits.conf file, but only the hard one is.
What am I doing wrong?
I'm running Amazon Linux on AWS EC2.
bash-4.1$ cat /etc/*-release Amazon Linux AMI release 2012.09