Issue with load balancer

Question

I am trying to load balance between two Apache http webservers. I created a load balancer using mod_proxy_balance from Apache.

When I run my client (which generates ~100s of threads every second and tries to browse through the website), after a few seconds I start getting 502 and 503 error response codes for few threads. There are few other threads that function properly.

I feel the loadbalancer might be overloaded but im not exactly sure. Can someone please let me know how to proceed in solving this issue? Thanks!

My proxy balancer setup:

<IfModule mod_proxy_balancer.c>

    <Proxy balancer://hgcc13.org>
        BalancerMember http://hgccvm90.org/ loadfactor=1
        BalancerMember http://hgccvm92.org/ loadfactor=1
    </Proxy>

    <Location />
        ProxyPass      balancer://hgcc13.org/
    </Location>
</IfModule>

When I saw the error_log in the proxy , First I find

" connection time out to the backend servers" , then i see

" ap_proxy_connect_backend disabling worker" , and then

" All workers are in error state",

The above 3 messages are repeated for a short amount of time. Then I find only

"do_ypcall: clnt_call: RPC: Unable to send; errno = Operation not permitted ".

I ran the experiment only for 5 minutes

Answer 1

I suspect an issue with the backend servers, not the proxy itself.

The 502 errors are indicative of a lack of response from the balancer members.

The 503 errors indicate that one of the Apache services has hit its MaxClients setting - this could be the proxy, but if it is, then I suspect that's only because it's waiting on other requests to come back from the unresponsive backends.

Verify which system is hitting its limit by checking for "server reached MaxClients" in the Apache error log on each system.

You may need to adjust the timeout on the balancer members (the ProxyTimeout directive or the timeout=x setting on the BalancerMember lines), and/or the MaxClients (and related MPM settings) of the servers.