Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using HAProxy 1.4.18 with the following backend configuration

backend staging
  option httpchk HEAD /check.txt HTTP/1.0
  http-check disable-on-404
  default-server error-limit 1 on-error mark-down
  server staging01 x.x.x.x:80 check observe layer7
  server staging02 x.x.x.x:80 check observe layer7

The servers are running multiple applications on apache/passenger.

The combination of httpchk and disable-on-404 allows graceful shutdown and removing a server from the lb quite easily while still being able to access directly (ie for testing).

I'm trying to setup observe in order to disable a server when an application is not working. I've broken the application configuration on staging02 so it always return a 500. It's correctly marked DOWN after the first 500 but then marked UP at the next httpchk.

Here's the log file:

Server staging/staging02 is DOWN, reason: Health analyze, info: "Detected 1 consecutive errors, last one was: Wrong http response". 1 active and 1 backup servers left. 2 sessions active, 0 requeued, 0 remaining in queue.
Server staging/staging02 is DOWN, reason: Health analyze, info: "Detected 1 consecutive errors, last one was: Wrong http response". 1 active and 1 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
Server staging/staging02 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 0ms. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue.

Is there a way to combine those two checks ?

share|improve this question
    
Just to be certain that HAProxy is definitely the problem, could you find the three matching log lines from your backend Apache server and add them to your question? –  Ladadadada Dec 7 '11 at 22:05
    
One of the apache server is purposedly misconfigured so that all applications on it return a 500 error code. However the /check.txt always work unless I remove the file. I want the apache server to be disabled if it returns 500 error even if the check.txt file is still there. –  ouranos Dec 7 '11 at 22:49
add comment

1 Answer

The distinction I understand now is that /check.txt does actually return a 200 response but all requests to the application return a 500. HAProxy sees the 500s coming back from the proxied requests and takes the server out of the pool but then initiates its own check, receives a 200 and puts the server back in the pool.

The solution would be to do one of:

  1. Configure Apache, rather than the application, so that every request returns a 500 response, even the static file /check.txt.
  2. Change /check.txt into a Ruby app that contains just enough logic to choose between a 200 and a 500 response when appropriate.
  3. Set the inter value to something ridiculous like 3600. This should give you an hour to do your testing or (if the server went down on its own) figure out the problem and bring it back up.
  4. Set the inter value to something smaller like 60 but set the rise value to something higher like 60. This would also give you an hour before the server was added back to the pool. (Note, these two are listed last because they're probably very bad ideas.)
share|improve this answer
    
Yes that's what's happening. I think I'll have to get rid of the /check.txt and point the httpchk to one application. The check.txt file was handy to quickly and gracefully remove a server from all the HAProxy (3 of them in HA setup, only one active). It also seems that disable server doesn't persist between reload. –  ouranos Dec 7 '11 at 23:25
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.