2 weeks ago, I moved my website from AWS to another host.

Updated DNS etc.

I am still seeing hits on the old site. (not many , about 200 a day)

Is it reasonable to assume that after that amount of time, those hits are from web crawlers coming in through the Elastic load balance URL and are not users whose DNS has not updated?

I have noticed that typing the Load balance URL into google returns some results. Not exactly sure why its in google but it seems to be there.

  • 1
    Do your server access logs show the user-agent string? What does a reverse dns lookup on their IP address show? What kind of time outs did you have set on your domain info (I ask because I know of some who think "we aren't planning any changes for a year or more so i'm setting all our timeouts to 300 days" is a good idea) – ivanivan Apr 28 '17 at 21:45
  • The TTL is set to 60 minutes. I used whatsmydns.net to see what it thinks the DNS is doing, which all checked out ok. In the access logs (I did a tail on the last 1000 lines) and I only got "health checker", "Apache (internal dummy connection)","Googlebot","Baiduspider" and "zgrab" , the vast majority being health check entries. – Charlie Smith Apr 28 '17 at 22:15
  • 2
    Configure your old server to log the Host header, if it doesn't already... Those hits might not even be looking for your site. (Also... I very aggressively rate limit Baidu Spider -- it sends all kinds of pointless, garbage requests). Meanwhile, configure your new web servers to throw an error -- 404, 410, 503 -- when it sees a Host: header it doesn't expect. You don't want your content indexed under the ELB hostname. – Michael - sqlbot Apr 29 '17 at 0:13

I have never come across a public nameserver that caches records for more than two or three days, so I would say that your assumption sounds safe. If you want extra certainty, you can use a global DNS checking tool like https://dnschecker.org to see what various nameservers are responding with.

If you can find the ELB URL in a google search, it is highly likely that web crawlers are hitting that path and generating traffic. If your web server is set up to track the request URI that is getting forwarded through ELB, you may be able to see what URL the requests are coming from.

| improve this answer | |
  • I used the DNSchecker you mentioned and all came through with what I expected, there were a couple that came through with an X , but none showed the old information. – Charlie Smith Apr 28 '17 at 22:19
  • 2
    While I haven't seen DNS servers failing to respect TTLs I have seen devices like set-top boxes that cache IPs forever. This is definitely an issue with ELB IP's because you will eventually see somebody elses traffic if you look closely. – chicks Apr 28 '17 at 23:59

If you had a popular website, then you might see this traffic for weeks or months. I work for a company with a top-100 global website, and when we change an IP address or change a cache breaker we continually see traffic coming in for a very, very long time.

There are all sorts of badly behaved tools out there. Badly written scrapers by university students, badly written scrapers by professional search engines, badly written botnets.

There are also badly behaved recursive DNS resolvers, although these are becoming less common, and caching for > 14 days would be very, very unusual.

It's plausible that these 200 hits you're seeing are not even real people - especially if they're just sending the HEAD verb.

It is also possible that these are users who are hitting your old ELB URL directly for whatever reason - you've mentioned that the ELB is for some reason indexed by Google. You could always use the Google advanced search to see if it has any reference to 3rd party sites linking to that URL.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.