We intend to run several components to run a web site. Like a load balancer, a cache and a couple of web serserves. To avoid single-points-of-failure we intended to use Heartbeat/Corosync/Pacemaker and instead of one load balancer just run two of them. Now we have different choices how we set them up and I would like to hear your opinion which one works better:
Run pairs of servers in different clusters. E.g. "loadbalancer1" and "loadbalancer2". That way we wouldn't have two few servers to reach quorum if one of the servers failed. We could switch off the quorum feature though but would risk a split-brain situation. This doesn't sound perfect but the configuration is clear and simple and it would probably just work.
Run triples of servers in different clusters. Like "loadbalancer1" and "loadbalancer2" and "loadbalancer-quorum". That way we have an extra server that is otherwise doing nothing but at least we can reach quorum in case of a failure.
Run all servers in a single cluster. That way the cluster would consist of "loadbalancer1", "loadbalancer2", "cache1" and "cache2". Then we could use contraints to make sure that the load balancing software only runs on "loadbalancer1" or "loadbalancer2" and that the cache software only runs on "cache1" or "cache2". I could imagine that this gets confusing. But we we would have enough servers to always reach a quorum.
Do you have experience with such a setup? What should we do?