Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are facing a problem with inconsistent 502 errors and tracking down the reasons has been a very frustrating exercise. We can reproduce the problem by sending several simultaneous requests quickly. The problem is that several is only in the range of 10 to 20 within a 5 seconds (not a typo). So clearly this type of load should be handled easily.

We really like the Nginx + Tornado approach but are considering going to a more traditional (e.g. threading) approach because this problem has been very difficult to solve. I was wondering if you a) know how to fix this issue and b) how we can tracked down the culprit(s).

The log files simply identify there being a connection refused. We have the same problem as this post: How do I debug a HTTP 502 error?

But there is no answer provided on how to solve the problem so I'm hoping you can help because this may be a common issue with this type of setup.

Thanks in advance,

Paul

share|improve this question

migrated from stackoverflow.com Aug 1 '11 at 4:48

This question came from our site for professional and enthusiast programmers.

    
Have you tried the putting the same load on a single Tornado instance? I'm guessing you're using one frontend NGINX server mapping to multiple Tornado instances as reccomended in the manual but make sure all long database queries are async requests so as not to lock up the web threads –  sam Aug 1 '11 at 8:12
    
This does not sound like normal behavior so I'm guessing there is a config problem somewhere. Are they actually refused connections or is NGINX timing out on connection (Which would suggest the applications are blocking the web threads in Tornado), we run a server with 8 tornado threads behind NGINX that easily handles over 100 requests a second so even on a low spec server you should manage at least 10 rps –  sam Aug 1 '11 at 8:15
    
Thanks. We are looking into the database side right now. We use the SQL Alchemy pool for our connections to MySQL. We were thinking that since each request is handled in its own sub-process that the database connections would also operate within a sub-process which would prevent blocking. But that may not be the case. –  PlaidFan Aug 1 '11 at 14:39
    
You might have more luck on the TornadoWeb mailing list groups.google.com/group/python-tornado –  sam Aug 1 '11 at 14:44
    
I think we found the root cause of this issue. It appears that if the NGINX conf file has 4 servers listed but only one Tornado server is spun up at the time then it will raise 502 Errors. We can't identify a setting for number of failed server connection attempts but it appears that we were able to eliminate the problem by pairing the NGINX conf file server count with the Tornado server count. –  PlaidFan Aug 1 '11 at 15:56

1 Answer 1

By default nginx is not configured to retry connections to another upstream if one of them sends back a 502 error. You basically need to add this:

proxy_next_upstream error timeout http_502;

To your configuration. This will prevent the 502 errors from being sent directly back to the client and instead cause nginx to try and hunt for a better upstream. It will attempt all of the upstreams before failing back to the client according to this post:

http://forum.nginx.org/read.php?2,152071,152212

Here is more details on the configuration directive:

http://wiki.nginx.org/HttpProxyModule#proxy_next_upstream

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.