Hot swapping for Linux web/database servers

Question

Is there a way to perform the following under Linux:

1. There are two web servers, main and backup
2. There are two database servers (postgres), main and backup
3. Web Servers are in sync with each other, ie. configuration/content/applications are the same
4. Backup database is continuously synced up with main database.
5. If either of main servers goes down, it's being replaced with backup one on the fly.
6. When main database server goes back up, all the data from backup server is uploaded to it.

To make it more complex, an important detail is that the two servers are actually VPS and they are hosted in different countries by diferent providers, so none of the obvious solutions for servers in the same subnet/rack etc are going to work here.

Essentially, I need the hot swapping working automatically with no or minimal user intervention, if possible. Recovery procedure is preferably automatic but can include some manual steps.

Answer 1

http://www.linuxvirtualserver.org/

would allow you to put a load balancer (or better, two) in front of your production and backup servers.

You could use heartbeat alone which would allow a public IP to be served by two IPs. It would have to sense the primary or secondary going down, and then would send out an announcement to become that public IP.

You would have something like:

Public IP   1.2.3.4
Primary Server 1.2.3.5
Backup Server 1.2.3.6

Everyone would see the single IP which would be served virtually by whichever machine was 'active'

Answer 2

Further to the orignal question, an important detail is that the two servers are actually VPS and they are hosted in different countries by diferent providers, so none of the obvious solutions for servers in the same subnet/rack etc are going to work here.

Answer 3

You should probably look at DNS failover for the web tier and perhaps Tungsten Cluster for the DB tier.

Cheers

Answer 4

Look into haproxy or Nginx as the frontend load balancer. This will monitor and redirect traffic from your primary to your backup server if primary goes down or vice versa aka automatic fail-over.

For the data syncing between your database, look into multi-master replication.