Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a load balancer/reverse proxy (either nginx or cherokee, does not matter which) deployed on my server and it points to multiple background webserver processes (either gunicorn or cherrypy, does not matter which) in a round-robin set-up.

In order to minimize downtime, I have a webserver restart script that kills a specific webserver process (out of say 8 processes) and then starts it up immediately again; and then goes on to the next webserver process (kill it, then start it up again) so at any moment in time, there will always be at least 7 processes available for my reverse proxy to point to.

This is great; but is there a way to "perfect" this process such that I do not get 502 Bad gateway at all? 502 gateway happens when a user so happens to be on the site and using a webserver process that gets temporarily killed and revived.

Obviously, the reason why I need the restart script is for deployment of new python code in my python application (running on either gunicorn or cherrypy).

share|improve this question
add comment

4 Answers

up vote 3 down vote accepted

nginx should do this out of the box. proxy_next_upstream's default error timeout setting will pass a request on to the next member of an upstream block if the initial server chosen is unreachable.

share|improve this answer
add comment

Why are you killing gunicorn? Just send it a SIGHUP like any other well-behaved Unix process and it will happily reload without losing any connections.

share|improve this answer
add comment

If you use haproxy and configure a backend for each webserver process and also use health checks on those backends, it will detect when one is down and mark it as offline, balancing requests to the healthy ones instead.
A user connected to the restarting webserver should be rebalanced to a working one after he refreshes the page or initiates another request.

share|improve this answer
add comment

Use apache httpd instead.

It has an ultra-configurable reverse-proxying loadbalancer that supports member drain and weighted preferences, not to mention online reconfiguration through the balancer-manager interface.

share|improve this answer
    
That won't solve my problem whichever loadbalancer I use. The problem (if it even can be solved) is to somehow kill and revive my background webserver process (the processes behind the loadbalancer) so quickly that the user won't even find out. –  Calvin Cheng Jul 25 '12 at 8:47
    
You missed the point, then. –  adaptr Jul 25 '12 at 8:49
    
O... you mean that with apache httpd, it is possible to make a BalancerMember offline first, restart the background process and then put the BalancerMember online again? And while the BalancerMember is offline, the user is routed to another background process? –  Calvin Cheng Jul 25 '12 at 8:51
2  
...that is how it works, yes :) –  adaptr Jul 25 '12 at 9:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.