1

I am using nginx and I am always hit by web crawler if I am correct.

I tried to configure fail2ban but the IP address cannot be detected by fail2ban.

The reason that it is not detected because it seems that it is a legitimate visitor. Here's the sample log:

116.73.68.36 - - [19/Jul/2013:23:57:47 +0800] "GET /sites/default/files/download/rhenz23/it4cai.zip HTTP/1.1" 206 14628884 "http://www.mysite.com/php/5297/computer-aided-instruction.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/kariuki/institute.zip HTTP/1.1" 206 14510149 "http://www.mysite.com/php/5040/automatic-online-examination-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 16357796 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/tovi/online_lot_reservation_system.zip HTTP/1.1" 206 14850935 "http://www.mysite.com/php/4088/online-lot-reservation.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 17217908 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/sanbunna9/online_gues_house.zip HTTP/1.1" 206 17594389 "http://www.mysite.com/php/5235/online-guest-house.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:50 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 17070214 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/welmarie/online_product_reservation_system.zip HTTP/1.1" 206 15074810 "http://www.mysite.com/php/3969/online-product-reservation-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/mindgamez/system1_0.zip HTTP/1.1" 206 15232701 "http://www.mysite.com/php/4094/online-membership-and-billing-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 15555605 "http://www.mysite.com/php/4171/online-management-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:53 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 16379516 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:53 +0800] "GET /sites/default/files/download/carol_janine_crislyn/chmscnet_0.zip HTTP/1.1" 206 17671134 "http://www.mysite.com/php/4178/social-networking-site-chmscnet.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:59 +0800] "GET /sites/default/files/download/malyn30/socialnetworkingsite.zip HTTP/1.1" 206 16711108 "http://www.mysite.com/php/3971/sample-simple-social-networking-site.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:04 +0800] "GET /sites/default/files/download/Franziholic/franzdarylduetes.zip HTTP/1.1" 206 17718916 "http://www.mysite.com/php/5408/reyans-burger-online-ordering-system-using-php.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:07 +0800] "GET /sites/default/files/download/kariuki/institute.zip HTTP/1.1" 206 16876180 "http://www.mysite.com/php/5040/automatic-online-examination-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:08 +0800] "GET /sites/default/files/download/carol_janine_crislyn/chmscnet_0.zip HTTP/1.1" 206 16685045 "http://www.mysite.com/php/4178/social-networking-site-chmscnet.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:15 +0800] "GET /sites/default/files/download/jkev/psits_voting_system.zip HTTP/1.1" 206 17465518 "http://www.mysite.com/php/5442/drag-and-drop-voting-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:32 +0800] "GET /sites/default/files/download/may_ann/onlineschedulingsystem.zip HTTP/1.1" 206 12997278 "http://www.mysite.com/php/scheduling-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"

BTW, I tested my fail2ban settings using the following command and it is working fine.

fail2ban-regex /var/log/ispconfig/httpd/mysite.com/yesterday-access.log /etc/fail2ban/filter.d/apache-badbots.conf

In fact it detected two IP address with the following log:

5.9.23.42 - - [19/Jul/2013:07:52:26 +0800] "GET / HTTP/1.1" 200 11220 "-" "Mozilla/3.0 (compatible; Indy Library)"
124.122.67.67 - - [19/Jul/2013:02:43:30 +0800] "GET / HTTP/1.1" 200 59663 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"
124.122.67.67 - - [19/Jul/2013:02:43:40 +0800] "-" 400 0 "-" "-"

So how come fail2ban does not detect it? Is this really a bot?

I also use cloudflare to prevent this kind of bot but I need to manually block the IP address or change the setting to "I am under attacked". This works fine if I am always watching my server. But what if I'm away from my computer? So how can I prevent this crawler?

Please help.

Update:

Here's my jail.local settings:

[nginx-badbots]
enabled  = true
filter = apache-badbots
action = iptables-multiport[name=BadBots, port="http,https"]
logpath = /var/log/ispconfig/httpd/mysite.com/access.log
bantime = 86400 # 1 day
maxretry = 1
Improve this question
1
  • 1
    So what is your jail.conf for this?
    – Michael Hampton
    Jul 19, 2013 at 19:06

1 Answer 1

Reset to default
0

Like say Micheal Hampton, what is your jail.conf rules on fail2ban for this ?

Maybe this help.: http://codelog.climens.net/2011/02/13/using-fail2ban-with-nginx-in-debian/

Also you can use dos-deflate instead of fail2ban, and limit_req on nginx: http://wiki.nginx.org/HttpLimitReqModule

Improve this answer
2
  • That's the tutorial I used link. How can I use dos-deflate to limit the connection per IP address per seconds and ban it automatically if it exceed that amount? Exam if the connection exceeds 2 per second per IP then ban it automatically.
    – jaYPabs
    Jul 19, 2013 at 23:10
  • On config file you need put how many conections you want, umm 2 per second is litte, if use dos deflate you need put 30 because dos deflate dont check nginx logs, it check netstat conections, if you dont want more than 2 or 3 conecctions per ip to nginx use limit_req on nginx config is better way.
    – Skamasle
    Jul 28, 2013 at 20:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .