I'm new to complex network infrastructure but I'm quickly realizing that we're outgrowing our centralized load balanced cluster.
The way we have things setup right now means that in order to add a new group of worker nodes in a different data-center the following latency is added: User connects to load balancer in data-center A, load balancer sends a request to data-center B's worker node, data-center B starts processing the page and in doing so has to send a database query to datacenter A (will be resolved soon thanks to the help of a SQL cluster) data-center B's worker has to send the page back to the load balancer which has to send the page back to the client.
TL;DR far too much latency is caused by having a central load balancer.
That being said we need a method of decentralizing our infrastructure and my best idea at doing this would be to do something at a DNS level. Intelligent, location based DNS.
We're currently using the nameservers of our domain provider Lightspeed Gaming, and I understand that if we're looking to grow this doesn't sound like it'd continue to be a viable option. Would I be correct in assuming that we're going to need to setup our own dedicated nameservers, or are there any nameservers that we may be able to pay to use out there that can accomplish that functionality?
If I do need to get a dedicated nameserver set-up what should I use and where do I start?
Please note that an essential requirement of this setup woudl be to detect a failure in the HTTP worker node before sending a request to it (resolving DNS to that specific worker)