Too many open files with nginx, can't seem to raise limit

Question

The server is Ubuntu 13.04 (GNU/Linux 3.9.3-x86_64-linode33 x86_64).

nginx is nginx/1.2.6.

I've been working on this for an several hours now, so here's what I'm getting and here's what I've done.

tail -f /usr/local/nginx/logs/error.log
2013/06/18 21:35:03 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:04 [crit] 3427#0: accept4() failed (24: Too many open files)
2013/06/18 21:35:05 [crit] 3426#0: accept4() failed (24: Too many open files)

Nginx running:

geuis@localhost:~$ ps aux | grep nginx
root      3422  0.0  0.0  39292   380 ?        Ss   21:30   0:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody    3423  3.7 18.8 238128 190848 ?       S    21:30   0:13 nginx: worker process      
nobody    3424  3.8 19.0 236972 192336 ?       S    21:30   0:13 nginx: worker process      
nobody    3426  3.6 19.0 235492 192192 ?       S    21:30   0:13 nginx: worker process      
nobody    3427  3.7 19.0 236228 192432 ?       S    21:30   0:13 nginx: worker process      
nobody    3428  0.0  0.0  39444   468 ?        S    21:30   0:00 nginx: cache manager process

Modified soft/hard limits in /etc/security/limits.conf (settings from the end of the file)

root soft  nofile 65536
root hard  nofile 65536

www-data soft nofile 65536
www-data hard nofile 65536

nobody soft nofile 65536
nobody hard nofile 65536

A reading of the max files

cat /proc/sys/fs/file-max
500000

And in /etc/pam.d/common-session:

session required pam_limits.so

With this added and the server restarted for good measure, for nginx I count the soft/hard limits by getting the parent process's PID and:

cat /proc/<PID>/limits
Limit                     Soft Limit           Hard Limit           Units     
Max open files            1024                 4096                 files     

The parent process runs as 'root' and the 4 workers run as 'nobody'.

root      2765  0.0  0.0  39292   388 ?        Ss   00:03   0:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody    2766  3.3 17.8 235336 180564 ?       S    00:03   0:21 nginx: worker process      
nobody    2767  3.3 17.9 235432 181776 ?       S    00:03   0:21 nginx: worker process      
nobody    2769  3.4 17.9 236096 181524 ?       S    00:03   0:21 nginx: worker process      
nobody    2770  3.3 18.3 235288 185456 ?       S    00:03   0:21 nginx: worker process      
nobody    2771  0.0  0.0  39444   684 ?        S    00:03   0:00 nginx: cache manager process

I've tried everything I know how to do and have been able to get from Google. I cannot get the file limits for nginx to increase.

Help?

Answer 1

Add the following line to your nginx and restart the process:

worker_rlimit_nofile 30000;

This will allow the workers to take on more files. You can then verify with:

su - nobody
ulimit -Hn
ulimit -Sn

This should output the new hard/soft limits.

Reference

Answer 2

In Ubuntu, edit /etc/pam.d/su add or uncomment the line session required pam_limits.so

Also, in /etc/security/limits.conf make sure you have TABS between the characters and not spaces.

Answer 3

make sure you run the following command after editing those files

sysctl -p

Then restart nginx