1

We've recently begun having trouble with web-scraper/DDoS service 80legs taking down our servers a couple times per week due to their abusive crawling practices. Initially we were simply dropping in the following at the bottom of the affected sites' .htaccess files:

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} ^.*80legs
  RewriteRule .* - [F,L]
</IfModule>

However, it's getting to the point where we just need to block them at the server level across all servers.

According to the Apache docs this config is valid to place in the Server config section, aka httpd.conf, but doing this does not have an effect. Is there a particular approach we can take to block/deny/redirect requests based on User-Agent at the server level on an Apache server with Virtual Hosts enabled?

Note: it is not possible to block this at the firewall level because:

  • 80legs uses what is essentially an opt-in botnet to crawl pages. Their last "incident" involved 5250 unique IPs from approximately 900 different networks/IP blocks from around the world.
  • We do not currently have the ability to do deep-packet inspection.
Improve this question

2 Answers 2

Reset to default
0

According to http://www.80legs.com/spider.html their user-agent string is 008, not "80legs" that you used.

Additionally, they say that their crawler respects robots.txt file, so you should give that a try.

Update your robots.txt to contain:

User-agent: 008
Disallow: /
Improve this answer
7
  • They SAY that. But many postings online claim they do not always respect robots.txt
    – Grant
    Mar 7, 2014 at 22:46
  • @Grant Seriously? You downvoted my answer because the company is supposedly not doing something their official documentation states? And that's my fault and deserves my reputation to be removed? In addition to giving an answer regarding robots.txt I also corrected OP's wrong info they had in their rewrite rule.
    – Mxx
    Mar 7, 2014 at 22:50
  • 1. They frequently ignore robots.txt. 2. That's still not server-wide. 3. For clarity, I downvoted.
    – Sammitch
    Mar 7, 2014 at 23:17
  • +1: robots.txt should be first line of defence against overzealous crawlers - and Sammitch never mentioned whether he/she had implemented a robots.txt
    – symcbean
    Mar 8, 2014 at 0:56
  • @Mxx I wasn't the one who downvoted it. Just telling you what I read when I looked into this company.
    – Grant
    Mar 8, 2014 at 1:06
0

Sorry, but I don't know what you mean by we just need to block them at the server level across all servers if it is not block this at the firewall.

Indeed, that is exactly where I would block them. Using fail2ban.

Their last "incident" involved 5250 unique IPs from approximately 900 different networks/IP blocks from around the world

Doesn't matter - it's trivial to script the action to set the block to (say) an 8-bit network - or if you're feeling adventurous map out the ASN and block that. Using very long rule chains can impact performance (but a lot less than allowing the traffic through by the sound of things) but you just adjust the duration the ban to prevent this.

We do not currently have the ability to do deep-packet inspection

Not needed - you use Apache to handle the HTTP traffic and redirect to a script which triggers fail2ban to implement it's action.

Improve this answer
1
  • Or have fail2ban watch access log file for that user-agent string and then trigger action from there.
    – Mxx
    Mar 8, 2014 at 3:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .