12

[I posted this on the Nginx forums, but had no responses a week later, so trying here]

I am a Linux and Nginx novice, but have learnt enough to get it installed and running and working as a simple reverse proxy for two internal webservers. This has been running fine for months, but I recently started getting 500 errors.

Here is the recent output of /var/log/nginx/error.log (I have replaced our company name with "companyname.com" and replaced our public WAN IP address with

2020/02/10 15:17:49 [alert] 1069#1069: *1011 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web1.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web1.companyname.com"

2020/02/10 15:21:41 [alert] 1069#1069: *2022 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web2.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web2.companyname.com"

2020/02/10 15:33:28 [alert] 1084#1084: *19987 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web2.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web2.companyname.com"

2020/02/10 15:34:16 [alert] 1084#1084: *39974 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web1.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web1.companyname.com"

2020/02/10 15:50:30 [error] 1086#1086: *1 client intended to send too large body: 4294967295 bytes, client: 176.58.124.134, server: london.companyname.com, request: "GET /msdn.cpp HTTP/1.1", host: "<WANIP>"

2020/02/10 16:32:56 [alert] 1086#1086: *19989 socket() failed (24: Too many open files) while connecting to upstream, client: 10.10.10.1, server: web1.companyname.com, request: "GET / HTTP/1.0", upstream: "https://<WANIP>:443/", host: "web1.companyname.com"

I have added the following to the end of /etc/security/limits.conf

nginx soft nofile 10000
nginx hard nofile 30000

I have added the following to /etc/sysctl.conf

fs.file-max=70000

...And rebooted. However, I'm getting the same problem immediately after a reboot.

Interestingly the IP address that appears in the log "176.58.124.134" I don't recognise and a quick google search suggests this is an abusive IP address. I can block at the firewall, but I'm not sure that's the problem.

Any tips, suggestions are grealy appreciated. Thanks.

6
  • 2
    You update ulimits but did you change also worker_rlimit_nofile in nginx config file? Feb 17, 2020 at 11:33
  • Yes, when troubleshooting this last week I added the following line "worker_rlimit_nofile 20000;"
    – DDIT
    Feb 17, 2020 at 11:55
  • And did you restart the service? Feb 17, 2020 at 12:01
  • Yep. I have just restarted now to be sure. But otherwise the server has been rebooted a few times in the last week.
    – DDIT
    Feb 17, 2020 at 12:14
  • 1
    What's the output of ulimit -n ?
    – Kunal
    Feb 17, 2020 at 14:22

2 Answers 2

20

Your nginx server is not keeping up with the number of requests arriving. More precisely it lacks available file descriptors to open connections to the upstream.

These are regulated by three parameters:

  1. The per-system limit in /proc/sys/fs/file-max, which limits the maximum number of fds for the whole system. Don't change it, the default is high enough (800000 on my small server).
  2. The per-process hard limit (RLIMIT_NOFILE), which can be only set by root (or a process having CAP_SYS_RESOURCES). This is usually quite high (ulimit -Hn, around 1000000), so no need to increase it. If you want to increase it the pam_limit.so configuration /etc/security/limits.conf will not help you, as nginx is started by systemd (I have to guess, since you don't mention your distribution), which does not use PAM. You'll need to edit the nginx.service file instead:

    systemctl edit --full nginx.service
    

    and add the following line to the [Service] section:

    LimitNOFILE=your_limit
    
  3. The per-process soft limit. nginx can increase it itself with the directive mentioned by Romeo:

    worker_rlimit_nofile = your_limit;
    

Each limit can not be higher, than the ones in the previous points.

However, unless your website became extremely popular overnight, it is much more probable that it is experiencing a DDOS attack. You can mitigate it by limiting the number of connections per client using the http_limit_conn module. The example configuration in the documentation should directly apply to your case:

http {
    limit_conn_zone $binary_remote_addr zone=addr:10m;
    limit_conn addr 10;
    ...

which will limit the number of connections per IP address to 10. On most distributions you can put the two limit_* directives in a separate file (e.g. /etc/nginx/conf.d/limit.conf) without modifying the main nginx.conf.

6
  • Thanks. I added the two http_limit_conn lines you referred to into the nginx.conf, then restarted. I'm now getting a slightly different error when browing to the website "503: Service Temporarily Unavailable". error.log shows _2020/02/19 16:44:22 [error] 1091#1091: *23 limiting connections by zone "addr", client: 10.10.10.1, server: web1.companyname.com, request: "GET / HTTP/1.0", host: "web1.companyname.com"_ where did I go wrong? Thanks in advance.
    – DDIT
    Feb 19, 2020 at 16:57
  • Nothing, it is working as intended: limiting every IP address to 10 connections, afterwards it returns 503 errors. What's on 10.10.10.1? Is it another proxy or does it need to make many requests to your nginx? Feb 19, 2020 at 17:45
  • I see. Thanks. 10.10.10.1 is our firewall (gateway on Nginx server). By the way, the access.log shows around 20 connections in the last 2 hours, so its by no means a busy server. Around 10 of those are from 10.10.10.1, the rest are from external IPs which appear to be malicious, based on the URLs they are requesting.
    – DDIT
    Feb 20, 2020 at 10:10
  • 1
    You should probably configure the Nginx on the gateway to send the real IP of the client to the internal Nginx with: proxy_set_header X-Real-IP $remote_addr; (cf. proxy_set_header) on the gateway and set_real_ip_from 10.10.10.1; (cf. set_real_ip_from) on the internal Nginx. This way you'll have 10 connections per real IP and not 10 for all the requests proxied by 10.10.10.1. Feb 20, 2020 at 17:03
  • 2
    LimitNOFILE=70000 fix the issue
    – Hisham
    Dec 5, 2020 at 10:52
1

I saw this message in Nginx logs when I had the error in proxy configuration - by mistake I put Nginx port number in proxy_pass directive (instead of the port of the upstream services) and as a result any request sent to Nginx was looping through until the server ran out of workers due to too many opened connections which actually count as files. Another symptom of this mistake was a growing Nginx access log where each line had one more occurrence of the IP address than the previous one. The problem was gone after I fix the config file. I hope this helps someone.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .