Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I know that domain names are constantly changing, and I know there are a lot of them, but there is clearly a region of the domain name space which is stable. How would I go about getting a list, even a very big one?

Such a thing must logically exist, even if it is in a distributed form, because the web's DNS servers resolve names to IP addresses. So in theory if I could poll all the DNS servers in the world at a moment in time I would have the complete list of mapped names. Is there a practical way of doing that?

As an aside, does anyone have any good estimates of how many domain names exist at the moment?

share|improve this question

migrated from stackoverflow.com Apr 8 '10 at 14:17

This question came from our site for professional and enthusiast programmers.

2  
Some stats here: domaintools.com/internet-statistics 117M active domains. If you just need domains and not hosts, then your source data is WHOIS and not individual DNS servers. –  Joe Apr 8 '10 at 13:54
    
@Sam, in many ways, first, the list is bound to be so large that the only way to handle it is programmatically, second, domain names are of general relevance to programming, especially since most of us are writing code which operates on the web. I would respond, how is it not? –  Simon Apr 8 '10 at 14:06
    
Simon if you create an account here on Serverfault with the same details as your StackOverflow one then you can associate and link the two to re-claim your question here. –  GAThrawn Apr 8 '10 at 15:27

2 Answers 2

A starting place would be to download the DMoz directory, and start spidering from there.

share|improve this answer

Netcraft run regular surveys of servers/domains and publish a lot of stats and lists. Their web server survey could be a good place to start.

As far as I know their data is gathered from a combination of users with their toolbar installed, people running ad-hoc server uptime/stats searches and spidering.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.