Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Sorry for being unclear previously.

We have a vmware virtualized server instance that is our main production server. I stores a series of web based applications on close to a hundred unique top level domains. For serving web pages we use a LAMP stack. This server is running our primary and secondary dns servers (on two different ip address than that used for serving web content). And finally we also host our mail (pop and smtp) using exim (i believe).

Recently we've had issues causing our root fs to become read only, preventing apache2 or mysql connections and prevent incoming email. Essentially taking down the web presence and email for many thousands of clients. The nature of the issue (still undetermined by under control) did not affect the bind so dns was still resolving fine.

Since then, we have begun to mirror the production web sites and associated mysql databases onto a secondary server. This server is completely production ready.

My question is, what are recommended methods for a failover in the case that apache on our main production server fails (for what ever reason) to quickly, if not automatically, start forwarding traffic as seamlessly as possible to the secondary.

DNS round robining is undesirable for us since we do not wish to the load over two servers, in fact we only ever want the secondary to receive http requests in the case the main server is non-responsive. This is in part to the fact that our mirroring process is one way and changes to the secondary server would be reflected in the main server and even lost.

share|improve this question
    
This isn't related to your main question, but your read-only fs is likely caused by VMWare disk issues. I don't know what your VMWare infrastructure is, but we run VMWare ESX servers with shared SAN storage, and issues with the LUNs (even a small interruption) will cause this on Linux VMs. Depending on your infrastructure, you might even want to consider VMWare Fault Tolerance as a solution to your main question (communities.vmware.com/community/vmtn/vsphere/…). –  Paul Kroon Jan 29 '11 at 16:48
    
Thanks for the input. however we don't manage the vmware infrastructure. The issues appears to have been fixed by a series of fsck checks. We've since decided to ditch the main sever and invest in a series of smaller instances to separate the responsibilities. –  xzyfer Jan 29 '11 at 23:06

6 Answers 6

up vote 3 down vote accepted

DNS round robin is not recommended because:

1- Different servers may not be exposed to the same amount of requests. So, they will be loaded in an unevenly manner.

2- DNS load balancing does not take into account the server availability. The server DNs record will remain and may be used in case of failure.

3- DNS caching will make it even worse. You don't have control over the DNS caches of your clients and any intermediate DNS server in between. If you plan to make your TTL value samller, it may not work as expected. Look at this post. The accepted answer says that Many DNS server do not honor your TTL.

The recommended solution is to install a load balancer like HAProxy along with a high availability solution like heartbeat. This setup should be installed on two machines. If one goes down, the other will take over the VIP (by heartbeat). The running machine will take care of checking backend servers health and distributing the load (by haproxy).

EDIT:

If you want the servers to work in active-passive mode, you don't need a load balancer. You can install heartbeat with pacemaker to monitor the system resources such as apache, mysql, etc. The cluster can be configured to keep only one active server.

share|improve this answer
    
But what would be the process of redirecting or proxying the traffic to the passive server? Would it be recommendable to move bind onto a separate server with haproxy? That way a failure on the active server wont affect our dns or proxying capability? –  xzyfer Jan 28 '11 at 13:18
    
In the active-passive mode, you don't need HAproxy anymore. You said you don't need load balancing! You just need heartbeat and pacemaker to monitor servers resources. –  Khaled Jan 28 '11 at 13:23
    
Apologies, I'm not familier with the concepts of heartbeat and pacemaker. Or HAproxy for that matter. –  xzyfer Jan 28 '11 at 13:32
    
It's ok. We are here just trying to help each other :) –  Khaled Jan 28 '11 at 17:57

For redundancy within a single site, on a single Internet feed, you want to put clustered hardware on your front end, with a standby box ready to take over the IP address of a failed box. But you'll be out of action if your ISP has a failure, or your site loses power or suffers some other problem.

If you want protection against loss of a whole site, or loss of your ISP, then there are really only two options. One is to get your own BGP autonomous system number, and run your own BGP routes, with peering (well, paid transit) with several different ISPs. The smallest netblock you can do that with is a /24, so you'll need to have a netblock at least that big. You can then advertise different routes to a different site if your main site goes down.

Your other option, as you suggest, is round robin DNS. Some people advise against this on theoretical grounds, and there are problems with Windows Vista clients selecting addresses non-randomly, but it should work fine for redundancy, with the backup box just reverse-proxying traffic back to the main box unless the main box/site/Internet feed goes down.

share|improve this answer

After some research it turns out DNS round robining ... isn't desirable or recommended for this service.

Really? I'd be very interested to see the articles describing this.

You've not mentioned what OS this runs on - which has a lot of relevance to how the clustering is implemented, nor what software you currently use for DNS / how easy it is to change this.

If you can possibly aviod it, I'd strongly recommend using a replicated type cluster rather than a shared storage cluster - particularly where the data is not changing frequently.

Bind already provides for master/slave replication so distributing the data across multiple platforms - so it's just a matter of working out some way of routing the requests to an available server.

Similarly for MySQL.

Implementing the same for your webserver files is simple using rsync.

While round-robin failover is a little slow compared with other methods - its MASSIVELY more robust, simpler to admin and cheaper than other approaches. Without knowing your reasons for diskliking round-robin, its hard to suggest somethng which might be more acceptable. (The guys who wrote HA-Proxy recommend using at least 2 proxy servers with round-robin in front of a HA cluster).

I would suggest that you use (at least internally) virtual addresses the master and slave server - this simplifies the business of promoting the slave. You might even want to use virtual addresses for each service/cluster node. This makes it easy to set up a heartbeat tied to address failover - this won't be any faster than round-robin but ives marginal benefits during reduced service - assuming its implemented correctly and you can resolve split brain problems.

share|improve this answer
    
round robin may provide crude HA in browsers with a long delay to half the requests but that doesn't work in other apps. When used in front of HAProxy it is only there for basic load distribution, the assumption being you are using some sort of carp/vrrp protocol to make the end of both ips always available. –  JamesRyan Jan 28 '11 at 10:08
    
sorry for not clarifing, but a simple google of dns round robin will bring up many articles arguing (convincingly) against the faults with using dns round robining. You can also reference wikipedia: en.wikipedia.org/wiki/Round-robin_DNS –  xzyfer Jan 28 '11 at 12:36
    
xzyfer does mention using a LAMP stack, which means he's running Linux (and Apache, MySQL, and PHP). The problem with DNS round robin is that there is too much of a delay for it to work for failover. It could be hours, days, or weeks before a client sees the new server because even though you can specify a TTL, clients don't have to respect it. It is a pretty cheap and effective method of load balancing (although there are obviously much better methods), but it sounds like xzyfer needs instant failover, not load balancing. –  Paul Kroon Jan 29 '11 at 16:52

The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer.

UCARP allows a couple of hosts to share common virtual IP addresses in order to provide automatic failover

share|improve this answer

From what I gather, you are looking for a high-availability solution - i.e. when one server goes down, another server can take over.

What ThomK suggested is one way to do it except that the single point of failure will be the nginx box. Another thing that you can look at is using HAProxy (or even nginx) but with some sort of IP based fail-over as well.

You can get a lot of ideas from elsewhere.

share|improve this answer

Install nginx in front of Apache. If one Apache server is down, nginx will exclude it and serve data from another "workers".

So, setup should look like that

nginx -> worker #1 (Apache), worker #2, worker #3 etc.

Of course nginx should be installed on dedicated box. One problem you have to resolve - what if nginx will be down, but...

nginx website: http://nginx.org

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.