is it normal for NGINX to accumulate connections in writing state?

Question

This is a graph showing connection count from nginx's stab_status, the drops in graph are server restarts done manually.

I find this behaviour strange, looks like some leakage. I think there are connections hanging that should be closed, but they stay in writing state.

Is this normal? Should I take an action? If so, in which direction?

I tried changing nginx's keepalive, and then some kernel settings like this https://gist.github.com/perusio/2154235

But this behaviour hasn't change.

The server is a VPS with one processor available and 1024mb ram.

EDIT1: versions

nginx version: nginx/1.6.2 built by gcc 4.8.2 (Ubuntu 4.8.2-19ubuntu1) TLS SNI support enabled configure arguments: --add-module=/root/ngx_pagespeed-release-1.9.32.3-beta --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-pcre-jit --with-http_ssl_module --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_scgi_module --with-ipv6 --with-http_stub_status_module --sbin-path=/usr/sbin/nginx --with-http_spdy_module

Note that it runs ngx_pagespeed

EDIT2: More info

Nginx is working as a reverse proxy for uwsgi/django app. I'll monitor uwsgi and post the results to see better if uwsgi has something to do.

Answer 1

As described by the stub_status module documentation, a connection in the writing state means :

Writing

The current number of connections where nginx is writing the response back to the client.

So you should investigate on your server response times and check why it takes so long for your clients to get nginx's replies and close the connection afterwards : slow network, high data volumes, chunked transfer-encoding, websockets ...

Answer 2

If some of your servers use SPDY or HTTP/2, your problem has its roots in a defect in the integration of this protocols. You can have a look in nginx ticket #714.

The current status is:

There is at least one connection leak in HTTP/2 as fixed in nginx 1.11.3, and the fix is not yet available in 1.10.x. Please try 1.11.x instead.

Answer 3

This seems to be normal behavior, as there will always be clients who have an unreliable connection leaving the server's connections in a "sending" state. They aren't reading any more data but Nginx will still wait until the send_timeout before closing the connection.

These may also be attacks where the malicious clients deliberately do that to exhaust the server's available connections & file descriptors, so if this becomes a real issue you should try lower the value mentioned above.