Resolving host is consistently slow

Question

I'm trying to fetch a page programatically and it takes exactly 10 seconds to resolve the host, every time. On another machine it takes exactly 30 seconds. Both Linux.

My code is in Java but the problem is reproducable using wget:

time wget -d  --header "User-Agent:Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Ubuntu/10.10 Chromium/11.0.696.65 Chrome/11.0.696.65 Safari/534.24" http://www.sportsdirect.com

This hangs for 10 secs on:

Resolving www.sportsdirect.com... 86.17.5.250

We're running on Linux.

To confuse things further browsers on the same machine fetch the same page immediately.

Any clues?

Answer 1

This is because wget and lynx aren't caching the DNS results, and your browsers may even have IPv6 disabled.

The DNS for www.sportsdirect.com is broken, and times out on AAAA records.

You can use the -4 or --inet4-only option to wget to force IPv4 only.

$ time host -t A www.sportsdirect.com
www.sportsdirect.com is an alias for sd.gwcorp.sports-world.com.
sd.gwcorp.sports-world.com has address 86.17.5.250

real    0m0.009s
user    0m0.000s
sys 0m0.000s

$ time host -t AAAA www.sportsdirect.com
;; connection timed out; no servers could be reached

real    0m14.008s
user    0m0.000s
sys 0m0.000s

Answer 2

The browsers may be caching the DNS results. It might be worth trying the query directly to the configured DNS servers to see if they are the problem -- perhaps one of them is particularly slow or down. Take the IP address from the nameserver line in /etc/resolv.conf and run

dig +trace www.sportsdirect.com 8.8.8.8

(replace 8.8.8.8 with your DNS server's IP address)