Kyle hit the nail on the head referencing the post that identifies the CAP Theorem.
Ultimately, this is going to boil down to budget and resources. The best way to handle availability on the border is going to be with network protocols such as BGP. Making the network highly available is easier, as you do not have the worry about data integrity in most cases.
Using DNS round robin is a compromise solution and less robust but certainly viable as well.
Lower on the stack, you have the Web servers, which are easier to dynamically failover, as with anything that does not involve data storage.
On the backend, you can replicate MySQL either over the Internet or preferably over a private link. If over the Internet, at least use SSL. A VPN would be better. This is the trickiest part and something I'm working through still right now. If you do not care about data integrity, it's simple. If your product is read focused, you have more options, as it's less complicated.
What I keep coming back to is the following..
High availability and business continuity are two distinct things. A high availability environment is best established within the same facility on the same internal network, as the best case scenarios can be applied with minimal risk to data. Split brain is substantially less likely with a 3' serial cable for monitoring server status in addition to the ethernet link. With a disaster, often there will be manual steps involved and a SLA dictating the impacts and terms. If the main data center burnt to the ground, 30-minutes to get production back up does not sound too bad.
I could probably write a book on the subject, as there's a lot to it. You will likely have to compromise requirements based on your resources, which will have to be communicated to the business. This is not a simple request.
One simpler solution may to have two external DNS providers, where one slaves off of your server on one site and the other external off a DNS on your other site. That way, if either ISP failed, you would be able to change the incoming path using DNS.
One Web path on one site, one on another. Dual master on the backend with manual failover up top. This would be simple and not risk data integrity but would have to be manual to do this.
