0

My server is getting killed by Baiduspider and no matter what I put in my robots.txt file nothing happens. So temporarily I need to block as many of it's IP addresses as I can via iptables. I am getting the IP addresses via:

grep -ri Baidu /var/log/apache2/access.log | cut -f1 -d' ' | sort | uniq

And my iptables rules look like:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            ctstate RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ftp-data 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:www 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:snpp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:mysql 
LOG        all  --  anywhere             anywhere            limit: avg 5/min burst 5 LOG level debug prefix `iptables denied: ' 
DROP       all  --  anywhere             anywhere 

I'm planning on adding the new rules with something similar to:

for ip in $(grep -ri Baidu /var/log/apache2/access.log | cut -f1 -d' ' | sort | uniq); do iptables -A INPUT -s $ip -j DROP; done

But I don't think I can just add them. I think they need to be inserted in a specific location. Where do they need to be inserted to take effect?

3
  • 1
    In the INPUT chain. First entry.
    – mailq
    Dec 27, 2011 at 19:30
  • 1
    But why not block 119.63.192.0/21 with one rule?
    – mailq
    Dec 27, 2011 at 19:35
  • There were 5 different subnets that had user agents claiming to be "Baiduspider/2.0". I ended up blocking all of them, rather than doing it by individual IP address.
    – nachito
    Dec 27, 2011 at 19:43

1 Answer 1

1

Your first rule looks to be allowing everything, making all of the rules below it pointless (is this intended?).

You'd need to insert the rule above that for it to have any effect; change your -A INPUT to a -I INPUT.

You might also consider just blocking them in Apache based on user agent instead - a 403 response might get the message across better than a failed connection, and won't require the resources that you're using now to respond to those requests with content.

2
  • 1
    The question is too localized. I queried my logs right now and see thousands of forged Baidu requests whereas they don't originate from the Baidu IP range.
    – mailq
    Dec 27, 2011 at 19:42
  • No Shane, the first rule wasn't intended. I think someone added that a while back when they complained that the firewall wasn't working properly. I'm trying to block those requests as soon as possible as they don't seem to be legitimate, otherwise I would consider using Apache to do it.
    – nachito
    Dec 27, 2011 at 19:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .