22

This is one that I've never really known how to do, and it's been nagging me from time to time for years. I've read all I could find, including all the answers here. None of them gives a real answer to the question. So please read carefully before flagging a duplicate; it is not.

dig, host, nslookup... none of them seems to be able to get what I'm after.

At most I can get pointers like ec2-xxx-xxx-xxx-xxx.us-east-2.compute.amazonaws.com.

But if I use this online tool, I get exactly what I'm looking for: every domain that resolves to the IP address given (or a hostname). In this case, it's a freemium service, so it'll only list the first few, but it works. In the following image, I use three domains that I own and serve from a VPS. They are totally unrelated, but they all appear here:

Enter image description here

What black magic are they using? How can we replicate it?

Improve this question
2
  • 1
    Noted it doesn't work at all on IPv6 - too much data to index. And it also misses things that don't start in www.
    – Criggie
    Jun 5, 2019 at 1:07
  • @Criggie: That's probably not the reason. If they are not using rDNS, but just bruteforcing regular domain→IP records as Michael said, then the number of "possible" addresses cannot actually affect the process in any way. (After all, why would they index and store addresses which have no domains?) More likely they just didn't bother.
    – u1686_grawity
    Jun 5, 2019 at 9:02

1 Answer 1

Reset to default
47

It's brute force.

They have looked up the IP addresses of every domain name they can find, and then compiled the results into their own database.

Improve this answer
11
  • 3
    The only part that is unclear to me is how they find a list of every domain name as there isn't a central list anywhere. Edit: Looks like they don't have everything because my website that has been up for years is not listed
    – Qwertie
    Jun 5, 2019 at 4:38
  • 12
    @Qwertie: certificate transparency ledgers are a good place these days – practically everything that has a properly rooted TLS certificate will appear in there.
    – Ulrich Schwarz
    Jun 5, 2019 at 6:18
  • 3
    @Qwertie : The Centralized Zone Data Service ( czds.icann.org/en ) could be a good starting point as well - it gives access to a large number of 'zone files' of the various top level domains.
    – Richy B.
    Jun 5, 2019 at 9:43
  • 1
    @Qwertie Registrars are exactly a centralized list of all domains... the only issue is that it's not just one of them and also not all of them give easily access to their zone files so access might not be completely free or immediate.
    – Giacomo Alzetta
    Jun 5, 2019 at 12:03
  • 5
    @GiacomoAlzetta no, registrars at most have a list of domains they sponsor, not all domains in all TLDs. registries on the contrary obviously have a list of all the domains they maintain, which is available in gTLDs through open access to zonefiles like Richy B. said (but noting that not all registered domain is published so some will be missing). Some others TLDs, like .FR have "open data" initiative where you can get access to many things, including list of domains. Other registries publish daily "newly registered domain names list".
    – Patrick Mevzek
    Jun 5, 2019 at 15:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .