3

I'm considering options for changing a 'high availability' website that provides a service via an https api.

The current setup is:

  • Two self-contained VMs, from different cloud providers (AWS and RackSpace)
  • One DNS load-balancer: this is where the HA also comes in, the service monitors the two VMs and if one appears to be unavailable it directs all DNS queries to the other

If load-balancing was not a requirement, could we do without the load-balancer by simply co-locating the DNS servers on the two machines, each replying only with it's own address when queried by DNS. In this scenario, if one VM is down that will remove both the service and the DNS server that points at the service so no clients will be directed to the server that is down, is that correct?


edit for clarity:

we are happy with the less-than-perfect 'HA' we currently have, this question is specifically about whether the changes I'm thinking of will make things worse or not.

  • 1
    In theory this could work, but DNS caching would ruin it. – Nathan C Jul 4 '14 at 14:12
  • @Nathan DNS caching is an issue with the current setup, are you saying it would be worse? – user83664 Jul 4 '14 at 14:12
  • What Nathan said; DNS load-balancing is essentially completely unusable on the modern internet. – MadHatter supports Monica Jul 4 '14 at 14:12
  • @MadHatter is there an obvious alternative that doesn't involve IP sharing between locations (which is out of our budget range)? – user83664 Jul 4 '14 at 14:13
  • 1
    Then no, there's no way to do this in a geographically-distributed way without spending real money. High availability costs; I used to tell customers that you get 99% uptime for the cost of the basic server setup, but each successive 9 you want multiplies the budget by between two and ten times. If you have a real business need for that kind of high-availability, you should have budget to match, because you'll need it. – MadHatter supports Monica Jul 4 '14 at 14:22
8

The direct answer to your question is Yes, it will make it worse.

This is because one of your name servers not responding will cause resolve delays all the time for clients who attempt to resolve via the failed name server, whereas the current technique will only fail +- half the clients until you detect the VM is down + TTL seconds.

Generally name servers are cached for 48 hours so during the lower of your downtime or name server updates + 48 hours, your users will have a randomly slow experience.

Your current implementation is better unless your VM down detection is slow. For the period between the VM going down and you detecting it + TTL the proposed solution will actually be better. But I am assuming that that is so small a period of time as to be ignorable.

  • 1
    Sorry, Ian, but would you care to expand on some of those assertions? Re para 2, ISPs are well-known to ignore TTLs on A records; I've seen six week delays in honouring a new A record, in the worst case. Re para 3, I'm not aware of any reason why an NS record would be magically cached for 48 hours regardless of its TTL. And the OP isn't proposing to change the NS records at any time, so the TTL of same strikes me as irrelevant. – MadHatter supports Monica Jul 4 '14 at 15:08
  • 1
    Sure @MadHatter. The authoritative NS come from the root servers for the domain, and the root servers almost universally cache NS records for 172,800 seconds (48 hours). Which means that when somebody queries from an empty cache situation, they will query X.root-servers.net for example.com. The root servers will point them to X.gtld-servers.net (also TTL 172,800), who in turn will hand out the authoritative nameservers for example.com with TTL 172,800. These are the NS records I was speaking of in para 2, not the A records. The A records TTL is under the control of Jack Daniels. – Ian Macintosh Jul 4 '14 at 15:25
  • 1
    Fair enough, point taken and thank you for that. Nevertheless I repeat my comment above that the OP's plan doesn't involve changing the NS records at any time, so the TTL is immaterial. – MadHatter supports Monica Jul 4 '14 at 15:37
  • 1
    As for the client delay, I just sat on one of my nameservers and watched a recursing query go through where one of the resolved party's nameservers didn't respond, and my BIND tried a second one after 800ms; I don't think that's an intolerable client delay. – MadHatter supports Monica Jul 4 '14 at 15:43
  • 1
    Ian, may I congratulate you on doing an excellent and well-referenced job of arguing your corner? I am unpersuaded at this time, but you've given me a lot of food for thought, and I've learnt something - and for both of those, many thanks! I shall go read your other answers, and upvote them if they're as good as this one. – MadHatter supports Monica Jul 4 '14 at 21:08
4

If I understand you correctly, you propose to have both distributed servers be the listed nameservers for the domain in question, each with an authoritative zonefile that contains a single A record pointing to the local server for the hostname on which your HTTPS service runs.

If that's correct, then yes, I would expect this to work based on the round-robin / stochastic nature of requests to nameservers. If one server goes down, it won't be able to respond to queries with its own address, so clients should fairly quickly fail over to the other for the DNS lookup.

You say you understand and accept that DNS cacheing means that a downed server may break clients who have cached the lookup that points to that server, possibly for lengthy periods of time because of ISPs that don't honour short TTLs. If that's all so, then I don't see any obvious holes in your proposal.

I just wouldn't do it myself in a month of Sundays.

  • The new scheme breaks because of caching of where the name server is, and the old scheme breaks because of caching of where the web server is. Are these failure modes identical or (for example) are ISPs more likely to ignore TTLs on nameserver records? – user83664 Jul 4 '14 at 14:52
  • I don't see why the new scheme would break that way. DNS is designed to failover quickly to a second nameserver for a zone if the first is unresponsive. – MadHatter supports Monica Jul 4 '14 at 14:55
2

DNS load balancing used to be the "thing" to use, but with caching in common use today it's not as practical.

In Windows, you have the "Network Load Balancer" which can assist with what you want to do by using multicast over a private network. You can still do this across providers with a VPN between the two.

In Linux, you'd need to use something like haproxy or other package to accomplish load balancing. This produces challenges on their own, though.

Look at your options and decide which would be most cost effective for your needs. I can't speculate beyond the very popular options.

  • Thanks, I will look into haproxy. Just to note though, we are using DNS load balancing presently and it is fine for our needs - the question is really just about whether I'd be making things worse (in terms of HA, not load-balancing) with the change I'm suggesting – user83664 Jul 4 '14 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy