I need to check 200kk (200,000,000) domains for availability and CMS information.
I'm using php 7.1 and simulating multi-thread check.
Hardware and config
- The server hardware: Multicore CPU, 64GB RAM, SSD disks, 500 Mbits dedicated bandwidth (OVH server).
- In resolv.conf there are Google DNS: 126.96.36.199 / 188.8.131.52
- ulimit -n is set to 655350
- nload to measure bandwidth load
I have checked the first 1,000,000 domains from the database using a different amount of parallel threads. Once I did so I faced the problem, that with increasing number of threads the number of domains, that do not respond within 30 seconds timeout are highly increased. Here are the results.
1. 1000 threads
TEST: 1,000,000 domains, 1000 parallel threads, 85 Mbits average bandwidth load, 1 hour check time. RESULT: 65% where successfully resolved, 35% were not resolved by timeout.
2. 300 threads
TEST: 1,000,000 domains, 300 parallel threads, 70 Mbits average bandwidth load, 2 hours check time. RESULT: 85% where successfully resolved, 15% were not resolved by timeout.
As we can see, with increasing the number of threads for 3 times, we do not get the bandwidth increase for 3 times more. We got a huge increase of domains, that were not resolved\accessed by timeout. Thus the speed of check was increased 2 times more.
Where is the bottleneck of the check? How can I utilize the full 500 Mbits bandwidth? Should I use my own DNS server, and if so, what are the correct config for that? Any ideas or help is highly appreciated.
Google DNS is probably rate-limiting you. Using your own recursive DNS resolver -- or several of them -- should be step 1.
I tried bind (DNS server) with its default configuration, without tuning it. I got HUGE bind load and the results were pretty the same.
Are there any ideas about how to tune bind to handle 200kk - 500kk requests per day? We can get another server just as a DNS server. Maybe we can use an alternative DNS server (not bind)?