Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a set of shared, static content that we serve up between our websites at http://sstatic.net. Unfortunately, this content is not currently load balanced at all -- it's served from a single server. If that server has problems, all the sites that rely on it are effectively down because the shared resources are essential shared javascript libraries and images.

We are looking at ways to load balance the static content on this server, to avoid the single server dependency.

I realize that round-robin DNS is, at best, a low end (some might even say ghetto) solution, but I can't help wondering -- is round robin DNS a "good enough" solution for basic load balancing of static content?

There is some discussion of this in the [dns] [load-balancing] tags, and I've read through some great posts on the topic.

I am aware of the common downsides of DNS load balancing through multiple round-robin A records:

  • there's typically no heartbeats or failure detection with DNS records, so if a given server in the rotation goes down, its A record must manually be removed from the DNS entries
  • the time to live (TTL) must necessarily be set quite low for this to work at all, since DNS entries are cached aggressively throughout the internet
  • the client computers are responsible for seeing that there are multiple A records and picking the correct one

But, is round robin DNS good enough as a starter, better than nothing, "while we research and implement better alternatives" form of load balancing for our static content? Or is DNS round robin pretty much worthless under any circumstances?

share|improve this question
1  
HAProxy not an option? –  Howiecamp Jan 9 '10 at 3:20
4  
as I said in the post, this is a specific question about this solution -- can we stay on topic? –  Jeff Atwood Jan 9 '10 at 3:43
    
load-balancing(en.wikipedia.org/wiki/Load_balancing_%28computing%29) is very different then redundancy(en.wikipedia.org/wiki/Redundancy_%28engineering%29). As Jeff stated in his opening paragraph, he's looking for a means of removing single point of failure(redundancy), not actual load-balancing. Can someone retag? –  antony.trupe Jan 9 '10 at 4:51
    
true, I wasn't precise.. I think of them in similar terms, but they're technically different. To me one implies the other -- can you even have load balancing without redundancy? –  Jeff Atwood Jan 9 '10 at 8:05
    
@jeff - absolutely, a dumb load balancer (which plain round robin DNS is) does not do redundancy. It's even harder if you're talking about balancing / redundancy across multiple sites. –  Alnitak Jan 9 '10 at 18:00

16 Answers 16

up vote 38 down vote accepted

Jeff, I disagree, load balancing does not imply redundancy, it's quite the opposite in fact. The more servers you have, the more likely you'll have a failure at a given instant. That's why redundancy IS mandatory when doing load balancing, but unfortunately there are a lot of solutions which only provide load balancing without performing any health check, resulting in a less reliable service.

DNS roundrobin is excellent to increase capacity, by distributing the load across multiple points (potentially geographically distributed). But it does not provide fail-over. You must first describe what type of failure you are trying to cover. A server failure must be covered locally using a standard IP address takeover mechanism (VRRP, CARP, ...). A switch failure is covered by resilient links on the server to two switches. A WAN link failure can be covered by a multi-link setup between you and your provider, using either a routing protocol or a layer2 solution (eg: multi-link PPP). A site failure should be covered by BGP : your IP addresses are replicated over multiple sites and you announce them to the net only where they are available.

From your question, it seems that you only need to provide a server fail-over solution, which is the easiest solution since it does not involve any hardware nor contract with any ISP. You just have to setup the appropriate software on your server for that, and it's by far the cheapest and most reliable solution.

You asked "what if an haproxy machine fails ?". It's the same. All people I know who use haproxy for load balancing and high availability have two machines and run either ucarp, keepalived or heartbeat on them to ensure that one of them is always available.

Hoping this helps!

share|improve this answer
1  
BTW you might be interested in an article I wrote about 4 years ago on these concepts : 1wt.eu/articles/2006_lb (take the PDF, reading the HTML through the pages is boring). –  Willy Tarreau Jan 9 '10 at 11:19
    
-1: "does not provide fail-over" - yes it does - and it implements it at the only place where non-availability can be reliably determined - at the client. –  symcbean Aug 26 '10 at 15:21
5  
Not at all. It would work if DNS did not make use of caches, but this is not the case and clients can't force caches to refresh. Talk to any person who regularly switches DNS entries and they'll tell you that eventhough they observe 80% switch in 5 minutes, it generally takes more than one week to get close to 100%. So DNS does not provide fail-over. –  Willy Tarreau Aug 28 '10 at 19:10
5  
A simple example of "load balancing without redundancy" is RAID0. –  robbyt Feb 26 '11 at 23:28

As load-balancing, it's ghetto but more-or-less effective. If you had one server that was falling over from the load, and wanted to spread it to multiple servers, that might be a good reason to do this, at least temporarily.

There are a number of valid criticisms of round-robin DNS as load "balancing," and I wouldn't recommend doing it for that other than as a short-term band-aid.

But you say your primary motivation is to avoid a single-server dependency. Without some automated way of taking dead servers out of rotation, it's not very valuable as a way of preventing downtime. (With an automated way of pulling servers from rotation and a short TTL, it becomes ghetto failover. Manually, it's not even that.)

If one of your two round-robined servers goes down, then 50% of your customers will get a failure. This is better than 100% failure with only one server, but almost any other solution that did real failover would be better than this.

If the probability of failure of one server is N, with two servers your probability is 2N. Without automated, fast failover, this scheme increases the probability that some of your users will experience failure.

If you plan to take the dead server out of rotation manually, you're limited by the speed with which you can do that and the DNS TTL. What if the server dies at 4 AM? The best part of true failover is getting to sleep through the night. You already use HAProxy, so you should be familiar with it. I strongly suggest using it, as HAProxy is designed for exactly this situation.

share|improve this answer
1  
totally off-topic, but we also have the problem of needing multiple HAProxy instances to fail over to -- what if the HAProxy machine fails? Subject of future questions, though, REALLY off topic for this one. –  Jeff Atwood Jan 9 '10 at 3:55
2  
+1 - The "With an automated way ... it becomes ghetto failover. Manually it's not even that." should be in big bold letters. DNS round-robin becomes a liability if you're not monitoring machines and removing them from the DNS if they fail, and the only reasonable way to do this is with an automated solution. There are much better solutions than DNS round-robin. –  Evan Anderson Jan 9 '10 at 4:56
    
totally agree, but 20% of your customers calling you with complaints is better than 100% of them calling with complaints.. –  Jeff Atwood Jan 9 '10 at 8:07
    
The key point (for me) that Schof makes in answering Jeff's question is that without fast failover Round Robin means that over time you have more customers impacted than without it but each (more frequent) incident impacts only a subset of customers rather than all. Whether this is "better" or not depends on the scenario but in most cases I would say it is not. –  Helvick Jan 10 '10 at 21:39
    
The best part of true failover is getting to sleep through the night. That is one clear definition! –  Basil Bourque Feb 6 at 10:23

I've said it several times before, and I'll say it again - if resiliency is the problem then DNS tricks are not the answer.

The best HA systems will allow your clients to keep using the exact same IP address for every request. This is the only way to ensure that clients don't even notice the failure.

So the fundamental rule is that true resilience requires IP routing level trickery. Use a load-balancer appliance, or OSPF "equal cost multi-path", or even VRRP.

DNS on the other hand is an addressing technology. It exists solely to map from one namespace to another. It was not designed to permit very short term dynamic changes to that mapping, and hence when you try to make such changes many clients will either not notice them, or at best will take a long time to notice them.

I would also say that since load isn't a problem for you, that you might just as well have another server ready to run as a hot standby. If you use dumb round-robin you have to proactively change your DNS records when something breaks, so you might just as well proactively flip the hot standby server into action and not change your DNS.

share|improve this answer

Round robin DNS is not what people think it is. As an author of DNS server software (namely, BIND) we get users who wonder why their round robin stops working as planned. They don't understand that even with a TTL of 0 seconds there will be some amount of caching out there, since some caches put a minimum time (often 30-300 seconds) no matter what.

Also, while your AUTH servers may do round robin, there is no guarantee the ones you care about -- the caches your users speak to -- will. In short, round robin doesn't guarantee any ordering from the client's point of view, only what your auth servers provide to a cache.

If you want real failover, DNS is but one step. It's not a bad idea to list more than one IP address for two different clusters, but I'd use other technology there (such as simple anycast) to do the actual load balancing. I personally despise hardware load balancing hardware which mucks with DNS as it usually gets it wrong. And don't forget DNSSEC is coming, so if you do choose something in this area ask your vendor what happens when you sign your zone.

share|improve this answer
    
and some DNS servers (or the control panels) are configured to give you a TTL of 7200 regardless of what you set it to - some large hosting companies do this IIRC. –  gbjbaanb Jan 9 '10 at 14:08

Windows Vista & Windows 7 implement client support for round robin differently as they backported the IPv6 address selection to IPv4. (RFC 3484)

So, if you have significant numbers of Vista, Windows 7, and Windows 2008 users, you're likely going to find behavior inconsistent to your planned thinking in your ersatz load balancing solution.

share|improve this answer
    
ah, thank you, excellent, I was looking for this link -- I had heard about this but couldn't find the reference! –  Jeff Atwood Jan 10 '10 at 10:08

I'm late to this thread, so my answer will probably just hover alone at the bottom, neglected, sniff sniff.

First off, the right answer to the question is not to answer the question, but to say:

  1. "You probably want Windows Network Load Balancing instead." OR
  2. "Get with the times, place your static content on something like Cloud Files or S3, and have a CDN mirror it worldwide."

NLB is mature, well suited to the task, and pretty easy to set up. Cloud solutions come with their own pros and cons, which are outside the scope of this question.

Question

is round robin DNS good enough as a starter, better than nothing, "while we research and implement better alternatives" form of load balancing for our static content?

Between, say, 2 or 3 static web servers? Yes, it is better than nothing, because there are DNS providers who will integrate DNS Round Robin with server health checks, and will temporarily remove dead servers from the DNS records. So in this way you get decent load distribution and some high availability; and it all takes less than 5 minutes to set up.

But the caveats outlined by others in this thread do apply:

  • Current Microsoft browsers cache DNS data for 30 minutes, so you're looking at more than 30 minutes failover time for a subset of your users, depending on their initial DNS cache state.
  • What the users sees during fail-over can be ... strange (you're not using auth on static content, and certainly not form auth, but the link shows something to watch out for).

Other solutions

HAProxy is fantastic, but since Stack Overflow is on the Microsoft technology stack, maybe using the Microsoft load balancing & high availability tools will have less admin overhead. Network Load Balancing takes care of one part of the problem, and Microsoft actually has a L7 HTTP reverse proxy / load balancer now.

I have never used ARR myself, but given that its on its second major release, and coming from Microsoft, I assume it has been tested well enough. It has easy to understand docs, here is one on how they see distribution of static and dynamic content on webnodes, and here is a piece on how to use ARR with NLB to achieve both load distribution and high availability.

share|improve this answer

I do not think it is a good enough solution because let's say you have two servers now and you round robin using DNS to each server's IP address. When one server goes down, the DNS servers have no knowledge that it went down and will continue to serve that IP address, as part of the RR process. Then 50% of your audience will get a broken site missing javascript or images.

Perhaps it is easier to point to a common IP address that is handled by Windows NLB representing two servers behind. Unless you are using a Linux server for your static content, if i remember reading that somewhere?

share|improve this answer
    
NLB is just round-robin at the server NICs, rather than at the DNS server. For this on Linux you want a HA solution - RedHat has one, or look at UltraMonkey for lots of detail. –  gbjbaanb Jan 9 '10 at 14:06
    
yeap i know what NLB does. I'm recommending that over DNS RR because a server failure won't cripple half the users. –  icelava Jan 9 '10 at 16:59
    
@gbjbaanb or put another way, NLB is round robin at Layer 2. DNS based round robin is at (or depends on) Layer 7 –  Alnitak Sep 7 '11 at 7:40

It has very marginal use, enough to get you by while you put a real solution in place. Like you say, the TTL's have to be set quite low. This has the side benefit, though, of pulling out a problematic machine from DNS while it's having issues. Say you have SvrA, SvrB and SvrC handing out your content and SvrA goes down. You pull it out of DNS and after the short time period defined by your low TTL, resolvers will figure out a different server (SvrB or SvrC) that are up. You get SvrA back online and put it back into DNS. A short downtime for some folks, none for others. Not great, but workable. The more static servers you put in the mix the less likely you will be to have majority groups of users down.

You certainly will not get the true balanced distribution that a real load balancing solution will provide due to the topology of the Internet. I'd still watch the load on all of the servers involved.

share|improve this answer
    
the content is 100% static so the load is negligible -- even on one server. It's mostly bandwidth. –  Jeff Atwood Jan 9 '10 at 3:17
1  
All out the same pipe? –  squillman Jan 9 '10 at 3:35

Round-robin load balancing only works when you are also in control of the DNS Zone so that you can change the list of servers and push it to the zone masters in a timely manner.

As mentioned in one of the other answers, the hidden evil of round-robin is DNS caching which can happen anywhere between your servers and the client which completely negate the small benefit of this solution. Even with DNS TTL set to a very low value you have little control over how long ISP's or even the client's DNS cache will keep the now-dead IP address active.

It's an improvement over a SPOF for sure, but only marginal. I would take a look at who ever is hosting your server and see what they have to offer, many have some sort of basic load balancer service they can provide.

You may as well have a single server with the static content duplicated in S3 and switch to the S3 CNAME when your primary goes down. You will end up with the same delay but without the multiple server cost.

share|improve this answer

This really depends on what you're talking about and how many servers you're rotating through. I once had a site that ran on several servers, and I used DNS round robin on that due to mainly my novice at the time, and it really wasn't a big issue. It wasn't a big issue because it didn't crash. It was a really stupid non-complicated system, so it held up, and had a pretty constant traffic level. If it did crash from traffic, it was during the day and something I could easily take care of. I'd say your static content qualifies as simple enough to not cause crashes on its own.

Outside of hardware failure etc., how stable has your server been? How "spikey" is your traffic on this content? Assuming straight up Apache or something and relatively flat traffic, it's not going to crash a lot, and I would say round-robin is "good enough".

I'm sure I'll get down voted because I'm not preaching a 100% HA solution, but that's not what you asked for. It comes down to what you're willing to accept as a solution vs. effort spent.

share|improve this answer

If you were using RR DNS for load balancing, it would be fine, but you aren't. You're using it to enable a redundant server, in which case it is not fine.

As a previous post said, you need something to detect heartbeat and stop hitting it until it comes back.

The good news is heartbeat is available really cheaply, either in switches or in Windows.

Dunno about other OSs but I assume it's there as well.

share|improve this answer

I suggest that you assign an additional IP address to each of your servers (in addition to the static IP that you use for, say, ssh), and you take that into the DNS pool. And then you use some software to switch around these IP addresses in case a server fails. Heartbeat or CARP can do that, for example, but there are other solutions out there.

This has the advantage that for the clients of your service, nothing has to change in the setup, and you don't have to worry about DNS caching or TTL, but you can still take advantage of the DNS round-robin "load balancing".

share|improve this answer

It'll probably do the job, especially if you can have multiple IPs on your static boxes. have one "serve static content" IP and one "manage machine" IP. If a box then goes down, you can either use an existing HA solution or manual intervention to bring the IP from the failed machine up on either one of the other "cluster members" or a completely new machine (depending on how fast it would be to get that up and running).

However, such a solution will have some small issues. The load balancing will not be anywhere close to perfect and if you're relying on manual intervention you may have outages for some visitors.

A hardware load balancer can probably do a better job of both sharing the load and providing "cluster uptime" than DNS round-robin will. On the flip side, that is one (or two, since ideally you have teh LBs in a HA cluster) pieces of hardware that will need buying, power and cooling and (possibly) some time to get acquainted with (if you do not already have dedicated load balancers).

share|improve this answer

To succinctly answer the question (is round robin DNS good enough as a starter, better than nothing, "while we research and implement better alternatives" form of load balancing for our static content?), I would say that it is better than nothing, but you should definitely continue to research other forms of load balancing.

share|improve this answer

When researching Windows Load Balancing several years ago, I saw a document that stated that Microsoft's web farm was configured as multiple load-balancing groups, with DNS round robin between them. Since you can have multiple DNS servers responding in each namespace, and since Microsoft's load balancing is self-healing, this provides both redundancy and load balancing.

Downside: you need at least 4 servers (2 servers x 2 groups).

Answering Jeff's comment on Schof's answer, is there a way to DNS round-robin between HAProxy servers?

share|improve this answer

I've read through all answers and one thing I didn't see is that most modern web browsers will try one of the alternative IP addresses if a server is not responding. If I remember correctly then Chrome will even try multiple IP addresses and continue with the server that responds first. So in my opinion DNS Round Robin Load balancing is always better then nothing.

BTW: I see DNS Round Robin more as simple load distribution solution.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.