I have these entries in /var/log/nginx/access.log: - - [22/Mar/2018:19:20:54 +0000] "GET / HTTP/1.0" 301 193 "-" "-" - - [22/Mar/2018:19:21:40 +0000] "GET / HTTP/1.0" 301 193 "-" "-" - - [22/Mar/2018:19:21:52 +0000] "GET / HTTP/1.0" 200 17582 "-" "-"

they look suspicious to me, but my knowledge of this area is limited. What's going on? There's even several others from 107.155.xxx.xxx

2 of the addresses appear on this site as reported: https://www.abuseipdb.com/check/. They are and, apparently Jacksonville, U.S. and China.

I've configured fail2ban as described here as I'm on Digital Ocean with Ubuntu 14.04, but nothing in the fail2ban log.


From what i can tell there is nothing to worry about in these lines. It is not uncommon that you see bots visiting your website and crawling for directories every minute. Take a look at "/etc/fail2ban/filter.d/botsearch-common.conf" and "/etc/fail2ban/filter.d/nginx-botsearch.conf" to see how fail2ban tries to prevent badbots crawling your website. You could also try to block bots by user-agent. This could be one approach that could be used: https://gist.github.com/hans2103/733b8eef30e89c759335017863bd721d

But my advice is to just ignore bots, if your configurations are clean you don't have to worry much about it.

| improve this answer | |

A simple GET of / is going to happen all day every day - along with plenty of other bot generated traffic. There's almost no sense in wasting resources doing anything about it if you're running a webserver that can be accessed publicly. It's literally what it's there for. The fact that some are responded to with 301 suggests you're using Virtual Hosts and that the request came in without a valid Host header, and that your Nginx setup is to redirect them to your default Virtual Host.

The reason fail2ban isn't doing anything is because none of the traffic it's seen has been detected as malicious. By default it will be watching out for script scanners, HTTP Auth brute forces and other malicious patterns.

| improve this answer | |

You can simply add this code to the beginning of your nginx code before the server tag to ignore the most common bots for your website.

map $http_user_agent $limit_bots {
     default 0;
     ~*(google|bing|yandex|msnbot) 1;
     ~*(AltaVista|Googlebot|Slurp|BlackWidow|Bot|ChinaClaw|Custo|DISCo|Download|Demon|eCatch|EirGrabber|EmailSiphon|EmailWolf|SuperHTTP|Surfbot|WebWhacker) 1;
     ~*(Express|WebPictures|ExtractorPro|EyeNetIE|FlashGet|GetRight|GetWeb!|Go!Zilla|Go-Ahead-Got-It|GrabNet|Grafula|HMView|Go!Zilla|Go-Ahead-Got-It) 1;
     ~*(rafula|HMView|HTTrack|Stripper|Sucker|Indy|InterGET|Ninja|JetCar|Spider|larbin|LeechFTP|Downloader|tool|Navroad|NearSite|NetAnts|tAkeOut|WWWOFFLE) 1;
     ~*(GrabNet|NetSpider|Vampire|NetZIP|Octopus|Offline|PageGrabber|Foto|pavuk|pcBrowser|RealDownload|ReGet|SiteSnagger|SmartDownload|SuperBot|WebSpider) 1;
     ~*(Teleport|VoidEYE|Collector|WebAuto|WebCopier|WebFetch|WebGo|WebLeacher|WebReaper|WebSauger|eXtractor|Quester|WebStripper|WebZIP|Wget|Widow|Zeus) 1;
     ~*(Twengabot|htmlparser|libwww|Python|perl|urllib|scan|Curl|email|PycURL|Pyth|PyQ|WebCollector|WebCopy|webcraw) 1;

And this to your / location

    location / {
            if ($limit_bots = 1) {
                    return 403;
| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.