Is it possible to block web crawler from downloading files (like zip file) in my server?
I supposed to create a PHP script using cookies to track visitors specially web crawlers to login/register after downloading 3 files. But I found out that web crawler can bypass cookies.
Is it possible to block web crawler? Or is there any other option that will hide the files from web crawler after it downloads up to 3 files?
I can easily create a PHP script using cookies to force visitors login/register, but how about web crawler?
By the way, I'm using nginx and drupal CMS. Just giving this info if this can help.
robots.txt
file.robots.txt
file. Techniques to identify the bad ones would fill a book.