Is Round-Robin DNS “good enough” for load balancing static content?

Question

We have a set of shared, static content that we serve up between our websites at http://sstatic.net. Unfortunately, this content is not currently load balanced at all -- it's served from a single server. If that server has problems, all the sites that rely on it are effectively down because the shared resources are essential shared javascript libraries and images.

We are looking at ways to load balance the static content on this server, to avoid the single server dependency.

I realize that round-robin DNS is, at best, a low end (some might even say ghetto) solution, but I can't help wondering -- is round robin DNS a "good enough" solution for basic load balancing of static content?

There is some discussion of this in the [dns] [load-balancing] tags, and I've read through some great posts on the topic.

I am aware of the common downsides of DNS load balancing through multiple round-robin A records:

• there's typically no heartbeats or failure detection with DNS records, so if a given server in the rotation goes down, its A record must manually be removed from the DNS entries
• the time to live (TTL) must necessarily be set quite low for this to work at all, since DNS entries are cached aggressively throughout the internet
• the client computers are responsible for seeing that there are multiple A records and picking the correct one

But, is round robin DNS good enough as a starter, better than nothing, "while we research and implement better alternatives" form of load balancing for our static content? Or is DNS round robin pretty much worthless under any circumstances?

Answer 1

Jeff, I disagree, load balancing does not imply redundancy, it's quite the opposite in fact. The more servers you have, the more likely you'll have a failure at a given instant. That's why redundancy IS mandatory when doing load balancing, but unfortunately there are a lot of solutions which only provide load balancing without performing any health check, resulting in a less reliable service.

DNS roundrobin is excellent to increase capacity, by distributing the load across multiple points (potentially geographically distributed). But it does not provide fail-over. You must first describe what type of failure you are trying to cover. A server failure must be covered locally using a standard IP address takeover mechanism (VRRP, CARP, ...). A switch failure is covered by resilient links on the server to two switches. A WAN link failure can be covered by a multi-link setup between you and your provider, using either a routing protocol or a layer2 solution (eg: multi-link PPP). A site failure should be covered by BGP : your IP addresses are replicated over multiple sites and you announce them to the net only where they are available.

From your question, it seems that you only need to provide a server fail-over solution, which is the easiest solution since it does not involve any hardware nor contract with any ISP. You just have to setup the appropriate software on your server for that, and it's by far the cheapest and most reliable solution.

You asked "what if an haproxy machine fails ?". It's the same. All people I know who use haproxy for load balancing and high availability have two machines and run either ucarp, keepalived or heartbeat on them to ensure that one of them is always available.

Hoping this helps!

Answer 2

As load-balancing, it's ghetto but more-or-less effective. If you had one server that was falling over from the load, and wanted to spread it to multiple servers, that might be a good reason to do this, at least temporarily.

There are a number of valid criticisms of round-robin DNS as load "balancing," and I wouldn't recommend doing it for that other than as a short-term band-aid.

But you say your primary motivation is to avoid a single-server dependency. Without some automated way of taking dead servers out of rotation, it's not very valuable as a way of preventing downtime. (With an automated way of pulling servers from rotation and a short TTL, it becomes ghetto failover. Manually, it's not even that.)

If one of your two round-robined servers goes down, then 50% of your customers will get a failure. This is better than 100% failure with only one server, but almost any other solution that did real failover would be better than this.

If the probability of failure of one server is N, with two servers your probability is 2N. Without automated, fast failover, this scheme increases the probability that some of your users will experience failure.

If you plan to take the dead server out of rotation manually, you're limited by the speed with which you can do that and the DNS TTL. What if the server dies at 4 AM? The best part of true failover is getting to sleep through the night. You already use HAProxy, so you should be familiar with it. I strongly suggest using it, as HAProxy is designed for exactly this situation.

Answer 3

I've said it several times before, and I'll say it again - if resiliency is the problem then DNS tricks are not the answer.

The best HA systems will allow your clients to keep using the exact same IP address for every request. This is the only way to ensure that clients don't even notice the failure.

So the fundamental rule is that true resilience requires IP routing level trickery. Use a load-balancer appliance, or OSPF "equal cost multi-path", or even VRRP.

DNS on the other hand is an addressing technology. It exists solely to map from one namespace to another. It was not designed to permit very short term dynamic changes to that mapping, and hence when you try to make such changes many clients will either not notice them, or at best will take a long time to notice them.

I would also say that since load isn't a problem for you, that you might just as well have another server ready to run as a hot standby. If you use dumb round-robin you have to proactively change your DNS records when something breaks, so you might just as well proactively flip the hot standby server into action and not change your DNS.

Answer 4

Round robin DNS is not what people think it is. As an author of DNS server software (namely, BIND) we get users who wonder why their round robin stops working as planned. They don't understand that even with a TTL of 0 seconds there will be some amount of caching out there, since some caches put a minimum time (often 30-300 seconds) no matter what.

Also, while your AUTH servers may do round robin, there is no guarantee the ones you care about -- the caches your users speak to -- will. In short, round robin doesn't guarantee any ordering from the client's point of view, only what your auth servers provide to a cache.

If you want real failover, DNS is but one step. It's not a bad idea to list more than one IP address for two different clusters, but I'd use other technology there (such as simple anycast) to do the actual load balancing. I personally despise hardware load balancing hardware which mucks with DNS as it usually gets it wrong. And don't forget DNSSEC is coming, so if you do choose something in this area ask your vendor what happens when you sign your zone.

Answer 5

Windows Vista & Windows 7 implement client support for round robin differently as they backported the IPv6 address selection to IPv4. (RFC 3484)

So, if you have significant numbers of Vista, Windows 7, and Windows 2008 users, you're likely going to find behavior inconsistent to your planned thinking in your ersatz load balancing solution.

Answer 6

I'm late to this thread, so my answer will probably just hover alone at the bottom, neglected, sniff sniff.

First off, the right answer to the question is not to answer the question, but to say:

1. "You probably want Windows Network Load Balancing instead." OR
2. "Get with the times, place your static content on something like Cloud Files or S3, and have a CDN mirror it worldwide."

NLB is mature, well suited to the task, and pretty easy to set up. Cloud solutions come with their own pros and cons, which are outside the scope of this question.

Question

is round robin DNS good enough as a starter, better than nothing, "while we research and implement better alternatives" form of load balancing for our static content?

Between, say, 2 or 3 static web servers? Yes, it is better than nothing, because there are DNS providers who will integrate DNS Round Robin with server health checks, and will temporarily remove dead servers from the DNS records. So in this way you get decent load distribution and some high availability; and it all takes less than 5 minutes to set up.

But the caveats outlined by others in this thread do apply:

• Current Microsoft browsers cache DNS data for 30 minutes, so you're looking at more than 30 minutes failover time for a subset of your users, depending on their initial DNS cache state.
• What the users sees during fail-over can be ... strange (you're not using auth on static content, and certainly not form auth, but the link shows something to watch out for).

Other solutions

HAProxy is fantastic, but since Stack Overflow is on the Microsoft technology stack, maybe using the Microsoft load balancing & high availability tools will have less admin overhead. Network Load Balancing takes care of one part of the problem, and Microsoft actually has a L7 HTTP reverse proxy / load balancer now.

I have never used ARR myself, but given that its on its second major release, and coming from Microsoft, I assume it has been tested well enough. It has easy to understand docs, here is one on how they see distribution of static and dynamic content on webnodes, and here is a piece on how to use ARR with NLB to achieve both load distribution and high availability.

Answer 7

I do not think it is a good enough solution because let's say you have two servers now and you round robin using DNS to each server's IP address. When one server goes down, the DNS servers have no knowledge that it went down and will continue to serve that IP address, as part of the RR process. Then 50% of your audience will get a broken site missing javascript or images.

Perhaps it is easier to point to a common IP address that is handled by Windows NLB representing two servers behind. Unless you are using a Linux server for your static content, if i remember reading that somewhere?

Answer 8

It has very marginal use, enough to get you by while you put a real solution in place. Like you say, the TTL's have to be set quite low. This has the side benefit, though, of pulling out a problematic machine from DNS while it's having issues. Say you have SvrA, SvrB and SvrC handing out your content and SvrA goes down. You pull it out of DNS and after the short time period defined by your low TTL, resolvers will figure out a different server (SvrB or SvrC) that are up. You get SvrA back online and put it back into DNS. A short downtime for some folks, none for others. Not great, but workable. The more static servers you put in the mix the less likely you will be to have majority groups of users down.

You certainly will not get the true balanced distribution that a real load balancing solution will provide due to the topology of the Internet. I'd still watch the load on all of the servers involved.

Answer 9

Round-robin load balancing only works when you are also in control of the DNS Zone so that you can change the list of servers and push it to the zone masters in a timely manner.

As mentioned in one of the other answers, the hidden evil of round-robin is DNS caching which can happen anywhere between your servers and the client which completely negate the small benefit of this solution. Even with DNS TTL set to a very low value you have little control over how long ISP's or even the client's DNS cache will keep the now-dead IP address active.

It's an improvement over a SPOF for sure, but only marginal. I would take a look at who ever is hosting your server and see what they have to offer, many have some sort of basic load balancer service they can provide.

You may as well have a single server with the static content duplicated in S3 and switch to the S3 CNAME when your primary goes down. You will end up with the same delay but without the multiple server cost.

Answer 10

This really depends on what you're talking about and how many servers you're rotating through. I once had a site that ran on several servers, and I used DNS round robin on that due to mainly my novice at the time, and it really wasn't a big issue. It wasn't a big issue because it didn't crash. It was a really stupid non-complicated system, so it held up, and had a pretty constant traffic level. If it did crash from traffic, it was during the day and something I could easily take care of. I'd say your static content qualifies as simple enough to not cause crashes on its own.

Outside of hardware failure etc., how stable has your server been? How "spikey" is your traffic on this content? Assuming straight up Apache or something and relatively flat traffic, it's not going to crash a lot, and I would say round-robin is "good enough".

I'm sure I'll get down voted because I'm not preaching a 100% HA solution, but that's not what you asked for. It comes down to what you're willing to accept as a solution vs. effort spent.

Answer 11

If you were using RR DNS for load balancing, it would be fine, but you aren't. You're using it to enable a redundant server, in which case it is not fine.

As a previous post said, you need something to detect heartbeat and stop hitting it until it comes back.

The good news is heartbeat is available really cheaply, either in switches or in Windows.

Dunno about other OSs but I assume it's there as well.

Answer 12

I suggest that you assign an additional IP address to each of your servers (in addition to the static IP that you use for, say, ssh), and you take that into the DNS pool. And then you use some software to switch around these IP addresses in case a server fails. Heartbeat or CARP can do that, for example, but there are other solutions out there.

This has the advantage that for the clients of your service, nothing has to change in the setup, and you don't have to worry about DNS caching or TTL, but you can still take advantage of the DNS round-robin "load balancing".

Answer 13

It'll probably do the job, especially if you can have multiple IPs on your static boxes. have one "serve static content" IP and one "manage machine" IP. If a box then goes down, you can either use an existing HA solution or manual intervention to bring the IP from the failed machine up on either one of the other "cluster members" or a completely new machine (depending on how fast it would be to get that up and running).

However, such a solution will have some small issues. The load balancing will not be anywhere close to perfect and if you're relying on manual intervention you may have outages for some visitors.

A hardware load balancer can probably do a better job of both sharing the load and providing "cluster uptime" than DNS round-robin will. On the flip side, that is one (or two, since ideally you have teh LBs in a HA cluster) pieces of hardware that will need buying, power and cooling and (possibly) some time to get acquainted with (if you do not already have dedicated load balancers).

Answer 14

When researching Windows Load Balancing several years ago, I saw a document that stated that Microsoft's web farm was configured as multiple load-balancing groups, with DNS round robin between them. Since you can have multiple DNS servers responding in each namespace, and since Microsoft's load balancing is self-healing, this provides both redundancy and load balancing.

Downside: you need at least 4 servers (2 servers x 2 groups).

Answering Jeff's comment on Schof's answer, is there a way to DNS round-robin between HAProxy servers?

Answer 15

To succinctly answer the question (is round robin DNS good enough as a starter, better than nothing, "while we research and implement better alternatives" form of load balancing for our static content?), I would say that it is better than nothing, but you should definitely continue to research other forms of load balancing.