11

I'm having an issue with a certain individual who keeps scraping my site in an aggressive manner; wasting bandwidth and CPU resources. I've already implemented a system which tails my web server access logs, adds each new IP to a database, keeps track of the number of requests made from that IP, and then, if the same IP goes over a certain threshold of requests within a certain time period, it's blocked via iptables. It may sound elaborate, but as far as I know, there exists no pre-made solution designed to limit a certain IP to a certain amount of bandwidth/requests.

This works fine for most crawlers, but an extremely persistent individual is getting a new IP from his/her ISP pool each time they're blocked. I would like to block the ISP entirely, but don't know how to go about it.

Doing a whois on a few sample IPs, I can see that they all share the same "netname", "mnt-by", and "origin/AS". Is there a way I can query the ARIN/RIPE database for all subnets using the same mnt-by/AS/netname? If not, how else could I go about getting every IP belonging to this ISP?

Thanks.

2
  • 1
    Have you considered that the perpetrator might be using compromised machines rather than getting a new IP address each time? Jun 15, 2010 at 1:13
  • Does CloudFlare offer options for limiting bandwidth by user/IP? I haven't used them but I thought they did. That would be the easiest way, imo, just use a service to do the whole thing for you.
    – markspace
    Apr 4, 2017 at 18:23

5 Answers 5

8

whois [IP address] (or whois -a [IP Address]) will usually give you a CIDR mask or an address range that belongs to the company/provider in question, but parsing the results is left as an exercise for the reader (there are at least 2 common whois output formats).

Note that such wholesale blocking can also potentially knock out legitimate users. Before taking this approach you should contact the abuse desk at the ISP in question (usually listed in the whois information for their netblock or DNS domain, otherwise abuse@ is a good place to start) to see if the situation can be resolved diplomatically rather than technically.


Also note that there are some pre-made solutions to limit requests per second by IP - Check out mod-qos or your system's firewall/traffic shaping capibilities.

1
  • I know the whois output gives you an address range, but this ISP seems to have ranges all over the place. e.g. (these aren't the actual addresses by the way) the spider will come from 46.84.*.*, then 88.98.*.* and so on. There's no obvious pattern other than what was noted in my question (same AS and maintainer in whois). Contacting their abuse department will result in the emails being sent straight to /dev/null. It's a Chinese ISP. As for mod-qos? Limiting request per second is useless. The spider isn't THAT aggressive. I Can't see any obvious way to do what I want through iptables either.
    – user45795
    Jun 14, 2010 at 18:51
6

Figured it out on my own. Sort of.

robtex.com lists all announced IP ranges for a given AS at: http://www.robtex.com/as/as123.html#bgp

Still don't know how or where robtex retrieves this info from. If someone else wants to chime in and explain where the data comes from, that would be great.

6
  • 2
    it is grabbed from the BGP announcements that they see from a connected router. Jun 15, 2010 at 1:46
  • anonymous user has it - the only way I know of they could collect that data is to scrape BGP announcements & build a routing table with the AS numbers.
    – voretaq7
    Jun 15, 2010 at 14:42
  • I assume BGP announcements are not accessible to the general public?
    – user45795
    Jun 15, 2010 at 14:56
  • 1
    I think this site is broken now.
    – user161042
    Feb 21, 2013 at 6:00
  • 1
    this link seems to be broken now. Is there anywhere else which the same info is available? Jul 16, 2013 at 13:49
2

Since you have access to iptables, I will assume you have a root access on the system anyway. In this case, I would suggest instlling Fail2Ban which will just block an IP (for a certain time you decide) if they try to abuse a service (HTTP, DNS, Mail, SSH ..etc) by hitting the service port as N times within X period. (all users decided.)

I am using that on my server and I am getting very good results. specially with those chinease hackers wanting to hit into my SSH.

hit my home page for more information. I have a blog post all about fail2ban.

2

You can use Hurricane Electric's BGP Service.

If you have an IP address and want to know all address blocks registered to the same ASN, do this:

  1. Go to https://bgp.he.net and search for the IP address to get its ASN.
  2. Search for the ASN.
  3. View tables "Prefixes v4" and "Prefixes v6" for all address blocks.
1
  • That's a pretty good site. Thanks for the heads up! :)
    – tftd
    Nov 12, 2021 at 12:53
-2

You can try this tool. It is not fast, but working.

2

You must log in to answer this question.