up vote 2 down vote favorite
share [g+] share [fb]

I know that domain names are constantly changing, and I know there are a lot of them, but there is clearly a region of the domain name space which is stable. How would I go about getting a list, even a very big one?

Such a thing must logically exist, even if it is in a distributed form, because the web's DNS servers resolve names to IP addresses. So in theory if I could poll all the DNS servers in the world at a moment in time I would have the complete list of mapped names. Is there a practical way of doing that?

As an aside, does anyone have any good estimates of how many domain names exist at the moment?

link|improve this question
2  
Some stats here: domaintools.com/internet-statistics 117M active domains. If you just need domains and not hosts, then your source data is WHOIS and not individual DNS servers. – Joe Apr 8 '10 at 13:54
@Sam, in many ways, first, the list is bound to be so large that the only way to handle it is programmatically, second, domain names are of general relevance to programming, especially since most of us are writing code which operates on the web. I would respond, how is it not? – Simon Apr 8 '10 at 14:06
1  
@Simon, it's not programming related since it's not a programming question. I guess enough others agree with me since it got migrated (but I wasn't one of the ones that voted to close, I was waiting on your response to my comment). – Sam Apr 8 '10 at 14:20
Simon if you create an account here on Serverfault with the same details as your StackOverflow one then you can associate and link the two to re-claim your question here. – GAThrawn Apr 8 '10 at 15:27
feedback

migrated from stackoverflow.com Apr 8 '10 at 14:17

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 2 down vote

A starting place would be to download the DMoz directory, and start spidering from there.

link|improve this answer
feedback
up vote 0 down vote

Netcraft run regular surveys of servers/domains and publish a lot of stats and lists. Their web server survey could be a good place to start.

As far as I know their data is gathered from a combination of users with their toolbar installed, people running ad-hoc server uptime/stats searches and spidering.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard