Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We recently moved DNS provider because of poor speeds. When we moved to our new paid provider, we had no idea of what DNS query volume to expect.

We currently have an average of 3,000 visitors a day but our DNS query rate is closer to 18,000 a day. That's 6 times higher than the visitor rate.

The TTL for our domain was increased from one day to three days (four days ago) and this has had no impact at all.

I would be exceeding grateful if anyone has any ideas about what could be causing the volume of excessive queries?

Thanks

Ben

share|improve this question
    
I'd go with "don't worry about it". Unless you're seeing really high load, as in your DNS server is unhappy, just let it do the job. –  Bill Weiss Dec 22 '09 at 16:03
1  
Err. Did you actually bother to read the background to the question? I'll give you a clue - Paid DNS provider... –  ben Dec 22 '09 at 17:33
    
They charge you PER QUERY? Most paid DNS providers give you the first 100,000 in the package price... –  Mark Henderson Dec 22 '09 at 19:03
    
hahaha! No per thousand overage. First 200,000 in the package. At the current rate we'll be through that in 11 days. –  ben Dec 22 '09 at 20:09
    
Ahh ok. That's far more reasonable! –  Mark Henderson Dec 22 '09 at 20:41

6 Answers 6

DNS is not only used for HTTP protocol, it is also used for FTP / e-mail / etc . Probably other service, which requires name resolution, is causing load.

share|improve this answer
    
Thanks Kristaps. My only suspicion from the research I had done was focused on it being caused by another service. I had read somewhere that email spam can really push the load up. Thanks for re-inforcing those suspicions. All I have to do now is get my developer to respond before Christmas! As I am not very technical, I was just wondering whether there might be another explanation like being spidered by bots. The Googlebot crawls around 7,000 pages a day - I was wondering if other bots were pushing the load up. –  ben Dec 22 '09 at 13:26
    
Our other services TTL are currently lower - MX lookups are all set to 3600 and other A lookups are still set to 86400 (only the domain name seems set to 286400). CNAME, SOA and TXT are all set to 86400. –  ben Dec 22 '09 at 13:40

Running a network capture on my computer and filtering for DNS gives me these results:

Going to a web site, www.hp.com in my test, initiates 2 A record lookups from my computer for www.hp.com and several more lookups for additional FQDN "resources", such as welcome.hp-ww.com and met1.hp.com so it seems perfectly normal that DNS queries will outpace actual site visitors.

Increasing the TTL value is likely to help only for people that make frequent visits to your site. It has no bearing on visitors whos' DNS resolver caches don't have your DNS records already cached. Every visit from someone who hasn't been to your site within the TTL period will require a new DNS lookup. Also, even with a 3 day TTL, if someone who's visited your site has rebooted their computer or flushed their DNS cache in that 3 day period will likely require a new lookup to find your DNS records (dependent on their own DNS infrastructure).

My suggestion would be to run a network capture on your computer (flush your DNS cache first) and start a capture, filtering for DNS, and then visit your site. Stop the capture and take a look at it to see what DNS queries are issued from your machine. This will give you a good idea of the DNS queries being issued, the answers being returned, and may point out some misconfiguration in your DNS that you're unaware of.

share|improve this answer
    
Thanks Joe. I'll pass that on to our developer to try. Having done some research on it, I was under the impression that DNS queries should be well under actual traffic -but you are suggesting that isn't the case. –  ben Dec 22 '09 at 13:55
    
Yes. If you're strictly counting only unique visitors and not page hits, etc. then you may find that there are multiple DNS queries for every unique visitor to the site. The best way to get a feel for it would be to run a network capture on your computer and visit the site, then look at the DNS traffic from your computer that's related to the site to see what is being queried for. –  joeqwerty Dec 22 '09 at 14:36
    
Thanks Joe. If a visitor then visted a number of pages (our average is 4) could subsequent pages after the first page also incur further DNS lookups? –  ben Dec 22 '09 at 15:02
    
I guess it depends, if the site calls resources based on an FQDN (as in my HP example) and those FQDN's have not been resolved by the client (which they probably wouldn't be for new visitors) then that will cause additional DNS lookups on the client in order to resolve those FQDN's. Additionally, the TTL on those additional FQDN's could affect the number of DNS lookups. If the TTL for the www A record is three days, but the TTL on an additional FQDN is one hour, then a new DNS lookup will be performed for that FQDN after 1 hour, regardless of the TTL for the www record. –  joeqwerty Dec 22 '09 at 15:25
    
Great. Thanks Joe. I'll get that checked out too. –  ben Dec 22 '09 at 17:10

Kristaps has the right idea, there is more to DNS than just web-site lookups. Anything that even thinks about sending your domain email is going to generate a DNS request. Even stuff hoping to get lucky by attempting to mail 'roger@yourdomain.com', and damn the NDRs! People scanning for open SSH ports will also generate DNS traffic if you happen to have one on your resolveable IP addresses. These SSH scans are frequently 'outsourced' to botnets, so each connection attempt is coming from a different host that'll have to look you up. All of these generate DNS requests independent of your HTTP traffic.

share|improve this answer
    
Thanks Sysadmin. Thats really helpful enlargement on the previous point. I've got a feeling this is going to be one popular page in the search engines as there doesn't seem to be this kind of information out there publically at the moment on DNS query volumes vs real traffic - at least not that I could find (and understand). –  ben Dec 22 '09 at 20:07

As mentioned by other people, your basic assumption is wrong: there is little relationship between DNS queries and HTTP visits. On one hand, you can have more HTTP visitors than DNS requests (if the visitors come from the same network and share the same DNS cache, not even mentioning big caches like Google DNS), on the other, you can have more DNS requests than HTTP visits (things like spam, already mentioned, or the AAAA requests for the people who have IPv6).

So, you should not continue in this direction: measure the actual DNS request rate but do not try to deduce it from the number of HTTP visitors (which is, in itself, not a very reliable information).

share|improve this answer

Using Google Webmaster Tools it's possible to see how many DNS requests are done by Googlebot. On our webs it is something between 1000 and 5000 per day. The traffic does not seem to be related to the TTL, but rather to the number of pages you have and how often they change - that is - Googlebot activity. It looks like Googlebot instances, running on different IPs, do not use some common caching DNS server.

Use this simple script on your DNS server to see how many requests is doing Googlebot:

tshark dst port 53 | grep 66.249

On our NS I can see two requests per second. That's about the same rate that would be on a secondary name server.

share|improve this answer

What are the actual queries? Are they 100% A lookups for www.example.com?

Consider telling us the actual domain name, as we may be able to spot incorrect DNS data.

share|improve this answer
    
Hi Alex. We have requested a breakdown from the DNS provider yesterday - but nothing back yet. Hopefully that will provide a guide to the answer when it comes! Forgive me if we prefer to keep the domain private - we currently don't want our traffic stats publically known. –  ben Dec 22 '09 at 13:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.