I'm looking for a way to setup Apache as high-availability. The idea is to have a cluster of 2+ Apache servers serving the same websites. I can have the IP address of each server set up with round-robin DNS so that each request is randomly sent to one of the servers in the cluster (I'm not too concerned with load-balancing just yet, though that may come into play later on).
I already have it set up and working with multiple Apache VM servers (spread across multiple physical servers) serving websites, and round-robin DNS, and this works fine. The SQL database is set up using MariaDB in a high-availability cluster, the web data (HTML, JS, PHP scripts, images, other assets) are stored within LizardFS, and the sessions are stored in a shared location as well. This all works well until one of the servers in the cluster becomes inaccessible for whatever reason. Then a percentage of the requests (roughly the number of downed servers divided by the number of total servers in the cluster) are unanswered. Here are the options I've considered:
Automatic DNS Updates
Have some process that monitors the functionality of the web servers, and removes any downed servers from DNS. This has two issues:
First, even though we can set our TTL to some very low number (like 5 seconds), I've heard that a handful of DNS servers will enforce a minimum TTL higher than ours. And, some browsers (namely Chrome) will cache DNS for no less than 60 seconds regardless of TTL settings. So even though we're good on our end, some clients may not be able to reach sites for some time in the event of a DNS update.
Second, the program that monitors the functionality of the cluster
and updates DNS records becomes a new single point of failure. We
may be able to get around this by having more than one monitor spread across multiple
systems, because if they both detect a problem and they both make the same DNS changes, then that shouldn't cause any issues.
Make the IP addresses that are accessed and in round-robin DNS virtual, and have them reassigned to up servers from down servers in the case that a server goes down. For instance, server1's VIP is 192.168.0.101 and server2's VIP is 192.168.0.102. If server1 goes down, then 192.168.1.102 becomes an additional IP on server2. This has two issues:
First, to my knowledge, uCarp/Heartbeat monitors their peers specifically for inaccessibility, for instance, if the peer can't be pinged. When that happens, it takes over the IP of the downed peer. This is an issue because there are more reasons a web server may not be able to serve requests other than just being inaccessible on the network. Apache may have crashed, a config error may exist, or some other reason. I would want the criteria to be "the server isn't serving pages as required" rather than "the server isn't pingable". I don't think I can define that in uCarp/Heartbeat.
Second, this doesn't work across data centers, because each set of servers across data centers has different blocks of IP addresses. I can't have a virtual IP float between data centers. The requirement to function across data centers (yes, my distributed file system and database cluster are available across data centers) isn't required, but it would be a nice plus.
So, any thoughts on how to deal with this? Basically, the holy grail of high availability: No single points of failures (either in the server, load balancer, or the data center), and virtually no downtime in the event of a switch over.