2

I need to check 200kk (200,000,000) domains for availability and CMS information.

I'm using php 7.1 and simulating multi-thread check.

Hardware and config

  1. The server hardware: Multicore CPU, 64GB RAM, SSD disks, 500 Mbits dedicated bandwidth (OVH server).
  2. In resolv.conf there are Google DNS: 8.8.8.8 / 8.8.4.4
  3. ulimit -n is set to 655350
  4. nload to measure bandwidth load

Tests

I have checked the first 1,000,000 domains from the database using a different amount of parallel threads. Once I did so I faced the problem, that with increasing number of threads the number of domains, that do not respond within 30 seconds timeout are highly increased. Here are the results.

1. 1000 threads

TEST: 1,000,000 domains, 1000 parallel threads, 85 Mbits average bandwidth load, 1 hour check time. RESULT: 65% where successfully resolved, 35% were not resolved by timeout.

2. 300 threads

TEST: 1,000,000 domains, 300 parallel threads, 70 Mbits average bandwidth load, 2 hours check time. RESULT: 85% where successfully resolved, 15% were not resolved by timeout.

Summary

As we can see, with increasing the number of threads for 3 times, we do not get the bandwidth increase for 3 times more. We got a huge increase of domains, that were not resolved\accessed by timeout. Thus the speed of check was increased 2 times more.

The question

Where is the bottleneck of the check? How can I utilize the full 500 Mbits bandwidth? Should I use my own DNS server, and if so, what are the correct config for that? Any ideas or help is highly appreciated.

Update 1

Google DNS is probably rate-limiting you. Using your own recursive DNS resolver -- or several of them -- should be step 1.

I tried bind (DNS server) with its default configuration, without tuning it. I got HUGE bind load and the results were pretty the same.

Are there any ideas about how to tune bind to handle 200kk - 500kk requests per day? We can get another server just as a DNS server. Maybe we can use an alternative DNS server (not bind)?

3
  • 3
    Google DNS is probably rate-limiting you. Using your own recursive DNS resolver -- or several of them -- should be step 1. Mar 13 '19 at 17:28
  • 2
    Look at all metrics you have available from the OS and user space programs to find the bottleneck. Start with the basics: medium.com/netflix-techblog/… Consider profiling your code and visualizing it: daniellockyer.com/php-flame-graphs In this case, measure the request rates of DNS and HTTP. Mar 14 '19 at 12:30
  • "Maybe we can use an alternative DNS server (not bind)?" Look at unbound. Mar 15 '19 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.