4

System Configuration : Apache2, Django 1.10, Python 3, Ubuntu 16.04 LTS

Django debug=True.


/var/log/apache2/error.log

[52:53.057967] [wsgi:error] [pid 4303] [client 1.1.1.22:24409] Timeout when reading response headers from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466726] [wsgi:error] [pid 4305] [client 1.1.1.10:9787] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466729] [wsgi:error] [pid 4304] [client 1.1.1.4:18417] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466726] [wsgi:error] [pid 4307] [client 1.1.1.22:35116] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466756] [wsgi:error] [pid 4306] [client 1.1.1.22:19242] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467164] [wsgi:error] [pid 4336] [client 1.1.1.4:34187] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467212] [wsgi:error] [pid 4342] [client 1.1.1.22:28212] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py, referer: http://example.org/
[52:58.467282] [wsgi:error] [pid 4331] [client 1.1.1.22:31045] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467426] [wsgi:error] [pid 4341] [client 1.1.1.70:22784] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py, referer: http://example.org/

I do not know the cause of the error. But I have narrowed it down to the Django wsgi process. Since the server is hosting static files properly.

While Cloudflare sometimes shows 502 : Bad Gateway Error, the server itself shows 500 : Internal Server Error.

I have already tried restarting the server and checking Django's (debug) log file. There is no error information in the Django log files (at all).


How should I debug the problem? Since Django didn't log anything, I assume the problem could be caused in wsgi.

Note : The server was working fine earlier. I made some changes* (which are reverted back as-is); The Django shell works fine.

Changes*

  1. Installed django-pandas, django-model-utils, numpy, scikit-learn
  2. A program that utilises above libraries. (This change is reverted to original)

In other similar questions, the problem is caused during upload of a big file.

  • Could you share the relevant apache config? It sounds like either you might need to tune/check your LimitRequestBody settings, and/or the equivalent django settings. – iwaseatenbyagrue May 11 '17 at 17:27
  • @iwaseatenbyagrue please check my answer. – Suraj Thapar May 14 '17 at 14:12
4

The cause of the problem was numpy.

Python C extension modules, like numpy, are known to cause timeouts when used under mod_wsgi.

Source : Answer by Sean F on Timeout when reading response headers from daemon process

A similar question which I didn't find on initial search answered and explained by author of mod_wsgi says -

Some third party packages for Python which use C extension modules, and this includes scipy and numpy, will only work in the Python main interpreter and cannot be used in sub interpreters as mod_wsgi by default uses. The result can be thread deadlock, incorrect behaviour or processes crashes.

Source : Answer by Graham Dumpleton on Non-responsive apache + mod_wsgi after installing scipy

Solution

Add the below line to your httpd.conf. In my case the file was /etc/apache2/apache2.conf.

WSGIApplicationGroup %{GLOBAL}
  • Can you explain what change when you use "%{GLOBAL}"? – Nazkter Sep 29 '17 at 20:44
  • I get this error even with WSGIApplicationGroup %{GLOBAL} in my apache config. – Cerin Oct 27 '17 at 5:13
  • @Cerin did you make sure that that directive applies to all Python files being interpreted? (I.e. not buried in some Apache <Directory /> that limits the scope.) – Ezekiel Victor Nov 8 '17 at 22:34
2

I have received a similar error while trying to run opencv using mod_wsgi and apache. I guess the problem was probably multiple threads with underlying C code trying to acquire GIL and failing.

I solved it by setting threads=1 and processes=32(in my case it was appropriate) in WSGIDaemonProcess directive.

PS: Late to the party but thought it could help someone.

  • I see - so you can avoid thread deadlock by settings threads = 1. Can you please share where to make the changes (eg: which file)? Thanks – Suraj Thapar Jan 16 '18 at 19:02
2

As others have mentioned, this is caused by the process crashing for a number of potential reasons. One reason being that some Python dependency isn't thread safe.

If that's the problem, one work around is to switch Apache's MPM type. The prefork type doesn't use threads, so if the problem is numpy crashing due to thread misuse, then that should fix it. The worker and event types use less memory, but also use threads, so they can encounter this error.

To determine which type you're currently using, run:

apachectl -V | grep -i mpm

If you see Server MPM: prefork then you're already using prefork, meaning the cause of the error might be something else. If it says "worker" or "event", then you can switch to prefork by running:

sudo a2dismod mpm_event
sudo a2dismod mpm_worker
sudo a2enmod mpm_prefork
sudo service apache2 restart

Note, the main downside of prefork is that, since it's not using threads, it consumes more memory.

0

I got this from Python access to magic (libmagic). Some versions of magic behave like that. I just compiled a version and used that instead of the one that came with the OS, for some reason that solved it.

0

In my case I had to change the WSGIDaemonProcess line from:

WSGIDaemonProcess wsgi processes=2 threads=4 display-name=%{GROUP} \
  python-path=/var/www/appname:/var/www/appname/venv/lib/python2.7/site-packages user=wsgi group=wsgi \
  home=/var/www/appname

to:

WSGIDaemonProcess appname user=wsgi group=wsgi processes=2 threads=4 display-name=%{GROUP} home=/var/www/appname

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.