520 `recv() failed` only when running application as www-data

Question

I'm running a nginx proxy server that fronts a SignalR application with thousands of connected WebSocket clients.

To accommodate all of these connections, I have set the following in nginx.conf:

worker_rlimit_nofile 60000;
events {
        worker_connections 30000;
}

If I run my application up using the myname user account, everything operates as expected.

When I try to run the application using a systemd service definition with user www-data, the application starts fine, however, about 80% of the requests to the server/application now fail with (from nginx error.log):

nginx recv() failed (104: Connection reset by peer) while reading response header from upstream

Which results in 520 status codes (cloudflare) to the client application.

The other 20% of requests succeed as expected, so I know the application is running properly.

[Unit]
Description=MyApp

[Service]
WorkingDirectory=/home/myname/app/
ExecStart=/usr/bin/dotnet /home/myname/app/publish/App.dll
Restart=always
RestartSec=10
SyslogIdentifier=MyApp
User=www-data
EnvironmentFile=/myapp.env

[Install]
WantedBy=multi-user.target

I feel like I am hitting some limit that is below what I have specified with worker_rlimit_nofile 60000, and it is causing failures (though could be just a red-herring). Checking ulimit for both of these users shows that they have the same:

# su - myname -c 'ulimit -aH' -s '/bin/bash'
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31823
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31823
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

# su - www-data -c 'ulimit -aH' -s '/bin/bash'
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31823
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31823
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

What could be causing these failures when operating as www-data?

Answer 1

The resource limits you may have configured for logged in users (e.g. in limits.conf) don't apply to services started by systemd. These resource limits need to be configured in the service unit itself or by setting system-wide defaults in systemd-system.conf.

For example:

[Service]
LimitNOFILE=1048576