Found that our Apache load balancer didn't failover to the hot spares when the primary server's application was responding with a 500 error.
This is how we had the proxy setup
<Proxy balancer://hotspare> BalancerMember ajp://node1:8463 loadfactor=100 retry=30 BalancerMember ajp://node2:8463 status=+H retry=0 BalancerMember ajp://node3:8463 status=+H retry=0 ProxySet lbmethod=bytraffic </Proxy>
It only actually failed over when node1 was shutdown. Can anyone explain this behavour?