I have been experiencing performance issues on a website for a while, and it always seems to hit around the same time. Having analysed the logs I've found a big spike in requests which corresponds with this slowdown, with all requests coming from the same subnet. It feels to me like an attempt to scrape the site (it is a car hire site and the requests are sequential for each IP and with incremental search criteria) and I would like to identify the source.

The Subnet in question is 209.67.89.x which I can see is owned by Savvis however I can't reverse DNS any of the IPs - is there any other way I can gain more info on this (other than contacting them direct - I am also doing this)?

  • This smells more like a second tier search engine to me than malice... have you checked what is actually requested via that subnet`? – rackandboneman Jun 12 '12 at 14:49
  • The requests are all well formed GET requests from the search form however they haven't originated from the search form - no referrer, no prior visit, just the form submission URL. That's what leads me to believe it is screenscraping, I can't see a search engine doing that? – Macros Jun 13 '12 at 8:24

All you can do is to use whois to query the ARIN database and find the owner of the network.
Reverse DNS is not required to be setup.

Of course you can block that subnet on your servers or rate-limit them.


Robtex has a nice service for this. You will get the AS number, black listings (email, other if applicable). In your case this might be able to help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.