After suffering a recent outage, and reading the apache docs for mod_proxy http://httpd.apache.org/docs/2.2/mod/mod_proxy.html I think I understand the problem, and how to set some threshholds to address it, but wanted to verify.
The "max" option for proxypass defaults to "threadsperchild" for the mpm worker. THis would be logical in the case of one reverse proxy worker.
(from apache docs)
max 1...n Maximum number of connections that will be allowed to the backend server. The default for this limit is the number of threads per process in the active MPM. In the Prefork MPM, this is always 1, while with other MPMs it is controlled by the ThreadsPerChild directive
But the docs all use examples of a single proxy rule, and in our case we define multiple reverse proxies (I beleive all getting a dedicated worker, and all having an implied max of 256 connections per the documentation)
# issues -> JIRA ProxyPass /issues http://localhost:8080/issues ProxyPassReverse /issues http://localhost:8080/issues # wiki -> confluence ProxyPass /wiki http://localhost:8090/wiki ProxyPassReverse /wiki http://localhost:8090/wiki # source -> FEYE ProxyPass /source http://localhost:8075/source ProxyPassReverse /source http://localhost:8075/source # builds -> BAMBOO ProxyPass /builds http://localhost:8085/builds ProxyPassReverse /builds http://localhost:8085/builds # quality --> SONAR ProxyPass /sonar http://localhost:8092/sonar ProxyPassReverse /sonar http://localhost:8092/sonar
If I understand the documents that would mean that each one of these would be its own pool of "max" matching the "childsperthread"
So 5 proxies (workers), each at the default 256 means 1280 connections (max)
Now the general config for the apache servers is
<IfModule mpm_worker_module> ServerLimit 2 ThreadLimit 256 StartServers 2 MaxClients 512 MinSpareThreads 1 MaxSpareThreads 512 ThreadsPerChild 256 MaxMemFree 100 MaxRequestsPerChild 0 </IfModule>
So since there are only 2 processes allowing 256 threads we have a much smaller limit of 512 threads (which matches the number of maxclients, but does not support the backend hunger of 1280)
Would it therefore be advisable to spawn 5 or more server processes=, or alternately decrease the "max" on the proxies to ~100? OR am I completely wrong in my thinking.
The reason I ask all of this is because a Google Search APpliance recently creamed our server. The backend appservers where all fine(healthy metrics), but traffic through the webeserver totally froze up. (we are adding robots.txt rules, but I also want to address the performance bottleneck)
- Connection through mod_proxy would hang or timeout
- Connection through mod_jk (the 6th app not shown above) worked
- Static pages on the site may or may not have worked, did not manage to check duyring the outage.