3

I have nginx webserver.
I have a rich content site and i found that some malicious bots are trying to crawl my content. I blocked any curl or wget coming to my site like this

if ($http_user_agent ~* (curl|wget)) { return  301 $scheme://www.google.com/; }

but i found that i can access the content if i changed the --user-agent in the curl request like below

curl --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" http://example.com/mypage.php
OR
curl --user-agent "whatever" http://example.com/mypage.php

Any idea how could i block any request generated from curl or wget using Nginx regardless of the fake user-agent that has been sent

0

3 Answers 3

4

User-Agent

User-Agent can always be spoofed. There are other headers you can check, but more clever bots spoof those as well. e.g. Accept, Accept-Language, Connection and some others that are not always used per object type

Cookies

Lesser intelligent bots will not properly accept and send cookies, so you can protect some resources using cookies. This may have privacy implications that you should consider.

Javascript

Some bots are unable to process javascript. You can have a hidden javascript "puzzle" so to speak, that requires the browser compute the answer to a simple random math problem. This will break many API Restful Clients unless you find a clever way to exclude them.

Authentication

If you have resources you want to keep bots away from, then you will need to protect those resources with authentication.

Keep-Alive

If you are certain that everyone hitting your site will support keep-alive; including proxies, then you could block connections that do not support it. Some will find this option unorthadox.

Obscure options that may also limit browers and/or API clients

I have also found that some bots don't handle TLS1.2+SNI, as they are often using older libraries that do not support TLS1.2 and even less common, SNI. This will limit the ability of your users to hit API's on your site if that is relevant.

I will leave it to you to research how you might test and implement each of those things and which of them may or may not be appropriate. One size does not fit all.

3

Stolen from StackOverflow, you could put a URL in your website that is not accessible via a link, but when a crawler sees it, it would try to crawl this address. This would then allow you to block that crawler.

<a style="display:none" href="./dontgohere.php">A</a>

Once you have your IPs listed, you can use something ngx_http_access_module to block addresses, or even fail2ban. There are a few different solutions you could proceed forward with.

If you are blocking strictly on user agent strings, though, it won't be possible as Alexander says.

0

That is not possible. curl and wget could put any user-agent and there is no possibility to know who is crawling your site.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .