Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have been experiencing performance issues on a website for a while, and it always seems to hit around the same time. Having analysed the logs I've found a big spike in requests which corresponds with this slowdown, with all requests coming from the same subnet. It feels to me like an attempt to scrape the site (it is a car hire site and the requests are sequential for each IP and with incremental search criteria) and I would like to identify the source.

The Subnet in question is 209.67.89.x which I can see is owned by Savvis however I can't reverse DNS any of the IPs - is there any other way I can gain more info on this (other than contacting them direct - I am also doing this)?

share|improve this question
    
This smells more like a second tier search engine to me than malice... have you checked what is actually requested via that subnet`? –  rackandboneman Jun 12 '12 at 14:49
    
The requests are all well formed GET requests from the search form however they haven't originated from the search form - no referrer, no prior visit, just the form submission URL. That's what leads me to believe it is screenscraping, I can't see a search engine doing that? –  Macros Jun 13 '12 at 8:24

2 Answers 2

up vote 1 down vote accepted

All you can do is to use whois to query the ARIN database and find the owner of the network.
Reverse DNS is not required to be setup.

Of course you can block that subnet on your servers or rate-limit them.

share|improve this answer

Robtex has a nice service for this. You will get the AS number, black listings (email, other if applicable). In your case this might be able to help.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.