1

I'm hosting an application consisting of several Python modules. So far we use Apache in MPM worker mode and mod_wsgi, which is a working environment.

Now we want to check if nginx + uwsgi may be a more performant environment. Python is Python 2.6.6 and uwsgi is 2.0.7. I have the following uWSGI config for my application (reduced example):

[uwsgi]
chdir = /path/to/app
chmod-socket = 777
no-default-app = True
socket = /tmp/socket.sock
master = 1
processes = 4
threads = 2
enable-threads = true
touch-reload=/root/uwsgi.ini
manage-script-name = True
mount = /accounts=account.py
[... several more mount directives ...]
mount = /ping=ping.py
[... several more mount directives ...]
mount = /subscriptions=subscription.py
callable = application
enable-logging = 1
plugin = /usr/lib/uwsgi/stats_pusher_statsd
stats-push = statsd:graphite-int.cern.ch:8125:uwsgi-test
enable-metrics = 1
memory-report = 1
stats = /tmp/stats.sock

When I start uWSGI like this, at first everything looks good to me:

[uWSGI] getting INI configuration from rucio.wsgi.ini
*** Starting uWSGI 2.0.7 (64bit) on [Mon Aug 25 19:15:07 2014] ***
compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-4) on 22 August 2014 22:51:22
os: Linux-2.6.32-431.3.1.el6.x86_64 #1 SMP Mon Jan 6 11:34:51 CET 2014
nodename: rucio-server-dev-ngnix
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /root
detected binary path: /usr/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your processes number limit is 63837
your memory page size is 4096 bytes
detected max file descriptor number: 4096
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited UNIX address /tmp/rucio.sock fd 3
Python version: 2.6.6 (r266:84292, Jan 23 2014, 10:39:35)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]
Python main interpreter initialized at 0x26b1c00
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 415360 bytes (405 KB) for 8 cores
*** Operational MODE: preforking+threaded ***
initialized 108 metrics
mounting account.py on /accounts
WSGI app 0 (mountpoint='/accounts') ready in 1 seconds on interpreter 0x26b1c00 pid: 2839
[... several more mount outputs like the one above ...]
mounting ping.py on /ping
WSGI app 8 (mountpoint='/ping') ready in 2 seconds on interpreter 0x9b020b0 pid: 2839
[... several more mount outputs like the one above ...]
mounting trace.py on /traces
WSGI app 15 (mountpoint='/traces') ready in 2 seconds on interpreter 0xf393210 pid: 2839
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 2839)
spawned uWSGI worker 1 (pid: 2986, cores: 2)
spawned uWSGI worker 2 (pid: 2988, cores: 2)
spawned uWSGI worker 3 (pid: 2990, cores: 2)
spawned uWSGI worker 4 (pid: 2992, cores: 2)
metrics collector thread started
*** Stats server enabled on /tmp/rucio-stats.sock fd: 16 ***

But a soon as the first request comes in, independently of the requested mount point, I get the following Traceback (example is a GET /ping):

!!! uWSGI process 2988 got Segmentation Fault !!!
*** backtrace of 2988 ***
/usr/bin/uwsgi(uwsgi_backtrace+0x29) [0x46c8d9]
/usr/bin/uwsgi(uwsgi_segfault+0x21) [0x46ca61]
/lib64/libc.so.6() [0x38bf8329a0]
/usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x3a) [0x38c2c43c4a]
/usr/lib64/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43) [0x38c2ccfc93]
/usr/bin/uwsgi(python_call+0x1f) [0x47a0bf]
/usr/bin/uwsgi(uwsgi_request_wsgi+0x132) [0x47c602]
/usr/bin/uwsgi(wsgi_req_recv+0x92) [0x420352]
/usr/bin/uwsgi(simple_loop_run+0xc5) [0x464265]
/usr/bin/uwsgi(uwsgi_ignition+0x254) [0x468074]
/usr/bin/uwsgi(uwsgi_worker_run+0x330) [0x468400]
/usr/bin/uwsgi(uwsgi_run+0x3e5) [0x468865]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x38bf81ed1d]
/usr/bin/uwsgi() [0x41d189]
*** end of backtrace ***
DAMN ! worker 2 (pid: 2988) died :( trying respawn ...
Respawned uWSGI worker 2 (new pid: 3003)

What's I guess is the most interesting part, is that if I remove every other mount point, and for example keep only

mount = /ping=ping.py

everything works as expected. To make sure that this error isn't related to our application, I also tried the 'Hello World' Example provided on the uWSGI site. But the behaviour stayed the same. One mount point is working fine, anymore than one causes this segfault.

Also, as soon as I remove the threads setting threads = 2 (not the threads-enabled!) it also works fine with multiple mount points. I therefore suspect that the error is really restricted to having multiple mount points in multithreaded mode. And keep in mind, it works in Apache MPM worker mode, so I don't expect the application being the reason (at least not in this case :D )

Would really like to figure this one out, as it wouldn't be a fair comparison between the Apache nodes and the nginx/uwsgi nodes if only Apache is capable of multithreading, right?

If you need anymore information pleas let me know and will happily provide it. Thanks for any tips or ideas, and just for reading this far you're already an every-day forum hero ;-)

Cheers, Ralph

0

Try this patch

https://github.com/unbit/uwsgi/commit/e45f710694f18052aa00ee5bb866d9aeca76fb80#diff-d41d8cd98f00b204e9800998ecf8427e

but think twice before following such approach. Having multiple apps in the same process address space may looks cool but soon or later you will pay for a suboptimal implementation (too much corner cases and workarounds when using multiple threads with multiple interpreters in CPython)

If it worked well in mod_wsgi will very probably work with uWSGI too, but consider splitting each mountpoint in a dedicated process.

  • Hi Roberto, thanks for your quick response. The fix worked. uWSGI is now running with the same config as provided above. In your answer you also pointed out that this might not be the best way to go. What are the alternatives? I can think of two: 1) I use uWSGI in the emperor mode and defining only one application per uWSGI config. 2) I use supervisor and to start one uWSGI instance per config, again only defining one application. Are there more alternatives and what would you recommend? One more question: Will I still use the mountpoint directive in the config? – Ralph Aug 26 '14 at 8:55
  • Use The Emperor, and yes you will end mounting one app per vassal. Each app will be bound to a different socket, so you need to configure nginx to route requests to each of them. – roberto Aug 26 '14 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.