1

I'm running fairly simple PHP websites, on Ubuntu, with AWS EC2 instances, which use *.rds.amazonaws.com domain names to connect to their respective RDS instances.

This introduces 2 issues:

  1. When the DNS cache expires, every 5 seconds, the next request adds a minimum of 12ms to my websites processing time, which isn't ideal (I try to keep within a 100ms budget).

  2. Occasionally this resolution step fails, so rather than showing an error page to my customers, my script will sleep for ~500ms, and try again.


I'm wondering if there is something, like a local caching DNS resolver, that could handle these issues more gracefully?

Perhaps it could cache the IP address for 3 seconds, then automatically try to refresh that cache, knowing that it has 2 seconds left. And if it does continue to fail, still provide the old/stale response, as that's better than no response.

Or, it could offer up the cached response immediately, even if it has expired (it's probably still correct); and if did just provide an expired response, run an update, just incase the RDS instance has just moved.

Or, could I bodge it, with a script that checks the DNS response in a loop (dig +short), and when the IP address changes, update the /etc/hosts file, using a hostname such as database-abc.


As an aside, my sites tend to get about ~4 requests every 5 seconds during the day (so cache updates are fairly frequent), but there is low activity overnight.

  • When the DNS cache expires, every 5 seconds Why would anyone do that? That should be a DNS worst practice. A short TTL like that is begging for an outage. – Greg Askew Sep 22 '19 at 16:39
  • Hi Greg, the 5 second TTL is set by Amazon, for their RDS service. The idea is that, if one of the database servers was to fail, it can switch over to a different server by changing the DNS entry, and that would be updated in less than 5 seconds. – Craig Francis Sep 22 '19 at 16:49
  • Not surprised. Doesn't matter, it's still a worst practice. – Greg Askew Sep 23 '19 at 10:42
0

As per AWS Route 53 Documentation:

Amazon Route 53 effectively connects user requests to infrastructure running in AWS – such as Amazon EC2 instances, Elastic Load Balancing load balancers, or Amazon S3 buckets – and can also be used to route users to infrastructure outside of AWS. You can use Amazon Route 53 to configure DNS health checks to route traffic to healthy endpoints or to independently monitor the health of your application and its endpoints.

For your specific scenario, here is an article

  • EC2 is effectively using Route 53, it’s just that any network request adds a delay (12ms or more in my case), and a chance of failure (usually happens once every couple of days)... so I’m looking at what I can do on the server to avoid those issues, probably caching in some way. – Craig Francis Sep 22 '19 at 16:46
  • You should consider raising your TTL in the zone files. Any reason why they are set so low? Is your DNS changing often? – Oxymoron Sep 22 '19 at 16:49
  • This guy - youtube.com/watch?v=4ZtFk2dtqv0 - Nill, while he is kinda far out there, explains it very well... – Oxymoron Sep 22 '19 at 16:50
  • It’s set my Amazon, and I can’t find any way of changing it... even so, it has to be fairly low incase one of the servers was to fail (it happens quite often due to a bug they have at the moment), and whatever it’s set to, eventually a lookup needs to be done, and that will always add a delay, and a chance of failure. – Craig Francis Sep 22 '19 at 16:53
  • 1
    I don’t think load balancing helps... if you’re talking about EC2 instances, it’s still a single server that’s handing the request, and if it’s DNS lookup for the Database server fails, that still effects that request... and if you’re talking about the DNS server end, that’s more Amazons side of things (part of RDS), and that’s on the wrong end of the network connection (so still slow, and prone to errors). – Craig Francis Sep 22 '19 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.