Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm having an issue with a certain individual who keeps scraping my site in an aggressive manner; wasting bandwidth and CPU resources. I've already implemented a system which tails my web server access logs, adds each new IP to a database, keeps track of the number of requests made from that IP, and then, if the same IP goes over a certain threshold of requests within a certain time period, it's blocked via iptables. It may sound elaborate, but as far as I know, there exists no pre-made solution designed to limit a certain IP to a certain amount of bandwidth/requests.

This works fine for most crawlers, but an extremely persistent individual is getting a new IP from his/her ISP pool each time they're blocked. I would like to block the ISP entirely, but don't know how to go about it.

Doing a whois on a few sample IPs, I can see that they all share the same "netname", "mnt-by", and "origin/AS". Is there a way I can query the ARIN/RIPE database for all subnets using the same mnt-by/AS/netname? If not, how else could I go about getting every IP belonging to this ISP?


share|improve this question
Have you considered that the perpetrator might be using compromised machines rather than getting a new IP address each time? –  John Gardeniers Jun 15 '10 at 1:13
add comment

4 Answers

Figured it out on my own. Sort of.

robtex.com lists all announced IP ranges for a given AS at: http://www.robtex.com/as/as123.html#bgp

Still don't know how or where robtex retrieves this info from. If someone else wants to chime in and explain where the data comes from, that would be great.

share|improve this answer
it is grabbed from the BGP announcements that they see from a connected router. –  user6738237482 Jun 15 '10 at 1:46
anonymous user has it - the only way I know of they could collect that data is to scrape BGP announcements & build a routing table with the AS numbers. –  voretaq7 Jun 15 '10 at 14:42
I assume BGP announcements are not accessible to the general public? –  user45795 Jun 15 '10 at 14:56
Depends. There are looking glasses where that sort of thing can be found and anonymous route servers. Have a look at traceroute.org –  Vatine Jun 16 '10 at 9:04
I think this site is broken now. –  user161042 Feb 21 '13 at 6:00
show 1 more comment

Since you have access to iptables, I will assume you have a root access on the system anyway. In this case, I would suggest instlling Fail2Ban which will just block an IP (for a certain time you decide) if they try to abuse a service (HTTP, DNS, Mail, SSH ..etc) by hitting the service port as N times within X period. (all users decided.)

I am using that on my server and I am getting very good results. specially with those chinease hackers wanting to hit into my SSH.

hit my home page for more information. I have a blog post all about fail2ban.

share|improve this answer
add comment

whois [IP address] (or whois -a [IP Address]) will usually give you a CIDR mask or an address range that belongs to the company/provider in question, but parsing the results is left as an exercise for the reader (there are at least 2 common whois output formats).

Note that such wholesale blocking can also potentially knock out legitimate users. Before taking this approach you should contact the abuse desk at the ISP in question (usually listed in the whois information for their netblock or DNS domain, otherwise abuse@ is a good place to start) to see if the situation can be resolved diplomatically rather than technically.

Also note that there are some pre-made solutions to limit requests per second by IP - Check out mod-qos or your system's firewall/traffic shaping capibilities.

share|improve this answer
I know the whois output gives you an address range, but this ISP seems to have ranges all over the place. e.g. (these aren't the actual addresses by the way) the spider will come from 46.84.*.*, then 88.98.*.* and so on. There's no obvious pattern other than what was noted in my question (same AS and maintainer in whois). Contacting their abuse department will result in the emails being sent straight to /dev/null. It's a Chinese ISP. As for mod-qos? Limiting request per second is useless. The spider isn't THAT aggressive. I Can't see any obvious way to do what I want through iptables either. –  user45795 Jun 14 '10 at 18:51
add comment

You can try this tool. It is not fast, but working.

share|improve this answer
Welcome to Server Fault! Please read our faq in particular May I promote products or websites I am affiliated with here?. –  Iain Jan 23 '13 at 8:16
add comment

protected by Michael Hampton Jan 23 '13 at 16:47

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?