We are facing some unexpected behavior with round-robin load balancing on Apache when one of the Tomcat servers goes down.

Our Setup: we have 2 Apache web servers on the front end using mod_jk module for load balancing with round-robin load distribution. We have enabled session stickyness. The load is balance amongst 4 Tomcat servers on which the applications are running.

Sometimes under heavy load, if there is a slowness in our database tier eventually one of the Tomcat servers goes into a hung state and would need a restart. The moment we bounce the Tomcat server we see a spike in requests in one of the other Tomcat servers which then would also go into hung state and need a restart.

Eventually all the Tomcat servers hang in a similar fashion.

Why does the Apache transfer the whole load to one server instead of distributing the load?

We are now trying the worker.balancer.method=B to see if this helps to resolve our issue.

In the Images below we see that service threads shoot up,

  • in Server 1 when Server 4 goes down at about 11.50
  • in Server 2 when Server 1 goes down at about 11.55

enter image description here enter image description here enter image description here

  • Install a load balancer like varnish or f5 big-ip? – user130370 Aug 31 '12 at 8:02
  • Forgot to mention, we have a F5 before the apache's, but do you mean to say, use F5 instead of apache? That is one thing, also wanted to know if this would be the expected behavior in round-robin load balancing. – srayker Aug 31 '12 at 9:31
  • As you have only 4 tomcats, I'd fire up 2 more apache and configure each apache to access only tomcat : So all the loadblancing would be done by f5 big ip. – user130370 Sep 17 '12 at 22:05
  • I would like to know whether there is concrete information that F5 can handle the issue in a better way. As it is our production environment, there is little scope for trial based diagnosis. – srayker Sep 21 '12 at 4:29

(Posting an answer instead of a comment as it might be too long) :

I'm not saying "F5 can handle the issue in a better way" but :

  • I would prefer doing the load balancing task by load balancers : F5 big ip, among other products, where designed to do this job.
  • As you have a small setup (4 tomcat), I see no reason for now to have 2 levels of load balancing. Having only F5 checking that a custom jsp page returns 200 is imho much simpler.
  • The worst downside I remember are : When a node is down, some traffic is still directed to it until next health check (~ 5 sec by default iirc). Session are lost if a node is down (maybe tomcat offers the ability to have a workaround eg session in database...).

I don't think it's easy to find "neutral" public benchmarking/testing about the better equipement/software to do load-balancing. I can only advise you to do your own if you have spare f5 for stagging.

As a general rule, I'd make f5 do as much as they can : Load balancing, ssl certificates, url rewriting, asm, ... Not because f5 are better but because it is convenient to have everything in the same place. Unfortunately, when http traffic start growing over a few hundreds MB, you have to start making some job done by apache instead of f5.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.