0

My Django site is hosted under Nginx/uWSGI. The site becomes unreachable from time to time for a period from few minutes to few hours. It just returns 500 after long waiting.

I can see harakiri messages in uWSGI log when this happens. Requests do not reach Django application (I've tried debugging). Instead I'm getting errors in Nginx log:

2016/12/03 01:18:40 [error] 1330#0: *363441461 upstream timed out (110: Connection timed out) while reading response header from upstream, client: <ip address>, server: site.com, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///var/run/uwsgi/app/site/socket", host: "site.com", referrer: "<page url>"
2016/12/03 01:19:27 [error] 1330#0: *363441461 upstream timed out (110: Connection timed out) while reading response header from upstream, client: <ip address>, server: site.com, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///var/run/uwsgi/app/site/socket", host: "site.com", referrer: "<page url>"
2016/12/03 01:20:15 [error] 1330#0: *363441461 upstream prematurely closed connection while reading response header from upstream, client: <ip address>, server: site.com, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///var/run/uwsgi/app/site/socket:", host: "site.com", referrer: "<page url>"

In uWSGI logs I see this kind of messages:

Sat Dec  3 01:19:00 2016 - HARAKIRI [core 2] <ip address> - GET / since 1480727890
Sat Dec  3 01:20:15 2016 - HARAKIRI [core 2] <ip address> - GET / since 1480727999
Sat Dec  3 01:20:32 2016 - HARAKIRI [core 0] <ip address> - GET / since 1480727937

Sat Dec  3 05:04:15 2016 - uwsgi_response_write_headers_do(): Broken pipe [core/writer.c line 238] during GET / (<ip address>)

It seems the uwsgi messages depend on harakiri value. I can't be sure because the problem is on the heavy loaded production server and I can't do enough experiments. I've set harakiri 15 the last time this pappened, and now I can't return previous value 40 because the site hangs up again.

Nginx settings:

uwsgi_read_timeout 30;
limit_conn addr 100;

location ^~ / {
    limit_conn session 5;
    limit_req zone=session_req burst=5;

    proxy_set_header X-Forwarded-Proto $scheme;
    uwsgi_pass unix:///var/run/uwsgi/app/site/socket;
    include uwsgi_params;
    uwsgi_param UWSGI_SCHEME $scheme;
}

uWSGI settings:

module = deploy.wsgi:application
master
processes = 10
threads = 5
listen = 3072
thunder-lock
cpu-affinity = 1

reload-on-rss = 200
harakiri = 15
reload-mercy = 20
py-autoreload = 0
vacuum

What is the problem and how do I fix this?

0

Any clues before the harakiri? perhaps you are hitting reload-on-rss too often? That being said, I've hit mysterious process locks with uwsgi in the past and we never figured out the issue, in the end, a good working solution was just to move to gunicorn. we never looked back.

  • No, I have not found any clues. uWSGI consumes about 80kB per process even when the problem happens, while reload-on-rss=200. I also have not found any reload on rss in the uwsgi log. Misterious things is what we got already with very old uWSGI version. So the first thing that I did was update, but this did not help. I'm thinking about alternatives now too, but still want to give uWSGI a chance. – raacer Dec 3 '16 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.