Apache2/fastcgi/php fastcgi/haproxy - graceful restart problems changing configuration settings

Question

I have an Apache2 with php fastcgi setup; I want to do config changes without a client noticing it (so no dropped/failing requests), so I put haproxy in front of it; so I have 2 Apache2 servers running on different IPs on the same server and Haproxy load balancing/failover running in front of them.

Both Apache's and Haproxy have Keepalive off. When I run /etc/init.d/apache2_1 restart or apache2ctl -k graceful -f /etc/apache2_1/apache2.conf while running ;

watch -n 1 lynx -dump -source http:// server /test.php (which has echo rand(1,100000); in it)

it still drops 1 request with a Bad Gateway. How can I prevent that from happening?

Basically if I could tell Haproxy to finish all connections (finish means just let them run) to server1 but send all new ones to server2 at a given point in time , it would work.

Also notice; I need php-cgi; with mod_php things are easier.

Edit; I tried this; http://www.iterasi.net/openviewer.aspx?sqrlitid=afatecr91eadnhiil9agpg however requests that are going on arent' passed to the backup, so some clients are still getting a Bad Gateway.

To answer Ben's comment;

With cgi graceful seems to kick off all ongoing connections; with mod_php graceful works fine; no downtime. Haproxy sees when apache is going down (goes yellow in the stats screen), but fastcgi just cuts the scripts off in mid-air;

Sat Aug 15 19:00:55 2009] [notice] mod_fcgid: process /var/www/gui/wsapi.php(13987) exit(communication error), terminated by calling exit(), return code: 0

[Sat Aug 15 19:00:55 2009] [notice] mod_fcgid: process 14012 going graceful shutdown, sending SIGTERM

[Sat Aug 15 19:00:56 2009] [notice] mod_fcgid: process /var/www/gui/index.php(13999) exit(communication error), terminated by calling exit(), return code: 0

A lot of clients are also getting;

[Sat Aug 15 19:05:42 2009] [error] [client xxx.xxx.xxx.xxx] Premature end of script headers: index.php

Another sign that there is nothing graceful about the shutdown.

Answer 1

You can tell haproxy to stop sending connections to a server. For that, you have to specify "http-check disable-on-404" in your haproxy config, and have your web server return 404 (and nothing else) to haproxy's health checks. If haproxy sees this, it will mark the server as "NOLB", meaning that it's excluded from the load balancing, but still finishes its connections, and still accepts persistent connections requesting it. That way, all new clients go to other servers.

The simplest way to do this is to check for an "alive" file, which, once removed, will cause a 404 to be returned (hence the choice of this return code). Of course, developping a more complete server-side application to respond to health-checks is better !