The bots tag has no wiki summary.
0
votes
1answer
33 views
Is there a chance to block images spiders / bots on dedicated servers without using robots.txt or .htaccess?
We know that we can block certain spiders from crawling websites pages using robots.txt or .htaccess or maybe via the Apache configuration File httpd.conf.
But that requires to edit may be a large ...
1
vote
1answer
191 views
Configure .htaccess to block bot indexing but allow browser directory listing/viewing
I know in order to show a directory listing of my files in a browser through .htaccess I can use
Options +Indexes
and to prevent Google and most bots from crawling my directory I can use
Options ...
1
vote
3answers
384 views
How to block this URL pattern in Varnish VCL?
My website is getting badly hit by spambots and scrappers, I am using Cloudflare but the problem still remains there. The problem is spambots accessing non-existing urls causing a lot of load to my ...
0
votes
2answers
160 views
Nginx 400 errors due to random encoded string starting with “\x” from random IP addresses
I assume these are some sort of bots, but would like to know what are they trying to do to my server.
The logs in questions are below and the IP address has been changed from the original.
...
-3
votes
1answer
50 views
Stop BPimage walker bot [closed]
How can i block all bots except a few including BPImage walker...it is creating a huge amount of load on web site and consuming lots of bandwidth
4
votes
1answer
307 views
Server 2008 adit logs show 1000's of Failure Logon Attempts to the `Admin` account
I was just going through the logs of one of my servers and I noticed there are a ton of failed logon attempts for username Administrator, owner and Admin. I assume that these are bots since there is ...
1
vote
1answer
395 views
How to stop bots using nginx?
I need to detect bots that act as authenticated users and saturate database's available connections by sending repetitive requests. I am wondering whether it is possible to stop dummy them using nginx ...
2
votes
2answers
198 views
Can apache forward/route requests based on http header (loadbalancing and bots)
We have a situation where bots comprise 40% of page requests for a slow-performing web application.
Short Term Relief
As immediate relief, I thought of doing the following:
Setting up another ...
2
votes
2answers
151 views
ethical/legal consideration when redirecting
A webcrawler has bought our site down twice, it ignores our robots.txt and we have had no reply from their customer services or support using both e-mail and twitter.
I have had to create url ...
0
votes
1answer
230 views
webcrawler bots load test my website and it fails the test
we run a commerical website with a relavitely few number of customer at any one time ~30 users .
Frequently a webcrawler such as google bot, bing bot, and 80legs will bring our site to a grinding ...
0
votes
1answer
197 views
I need to block this BOT or not by a firewall rule?
SCENARIO: Windows Server 2008, IIS 7, SQL Server 2005
I have a company website and i notice in the log this BOT (i think is a BOT) that continually hit some of my pages.
...
4
votes
4answers
486 views
How to prevent bots from trying to guess a link on my site
My logwatch report which I installed recently shows me this:
--------------------- httpd Begin ------------------------
0.78 MB transferred in 5864 responses (1xx 0, 2xx 4900, 3xx 0, 4xx 964, 5xx 0)
...
0
votes
1answer
103 views
Lots of bots making fake calls
I have a small VPS with Apache + php on which i host just a small blog.
I've been looking through its logs and seen tons of calls that (i guess) have the hostname faked. Here's a sample. Obviously ...
3
votes
8answers
616 views
Determining if a bot is scraping utility bill content and how to block it
I work for a local government that bills for water usage and garbage collection. I received a call today from a customer that a "New York company" called him asking for his customer number and pin to ...
1
vote
4answers
1k views
Netstat: How to know if it's Human or Bot/Spider/DDOS
I'm using the following command to check how many connections I have from a single ip
netstat -anp |grep ':80' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
If it's DDOS attack(500+ ...
8
votes
5answers
4k views
How are these 'bad bots' finding my closed webserver?
I've installed Apache a while ago, and a quick look at my access.log shows that all sorts of unknown IPs are connecting, mostly with a status code 403, 404, 400, 408. I have no idea how they're ...
0
votes
2answers
373 views
Methods to prevent malicious crawlers/scrapers and DDoS Attacks
From last couple of weeks I have been experiencing bot attacks on my site. Basically crawlers are running on the site at a high frequency rate resulting in load increase. This results in bandwidth ...
1
vote
2answers
816 views
Search Engine Bot - Large amount of hits
I've started tracking user-agent strings on a website at the start of each session. Looking at the data for this month so far I'm seeing on search engine bot that keeps coming up a lot..
Mozilla/5.0 ...
2
votes
2answers
1k views
Blocking by user-agent string in httpd.conf not effective
I'd like to block some spiders and bad bots by user-agent text string for all of my virtual hosts via httpd.conf but have yet to find success. Below are the contents of my http.conf file. Any ideas ...
1
vote
1answer
464 views
Strange request in access.log, how to block?
I am using nginx on my own server, and I noticed a few days ago some strange request in my access.log :
77.50.217.37 - - [19/Aug/2011:17:50:50 +0200] "GET http://images.google.com/ HTTP/1.1" 200 151 ...
0
votes
2answers
258 views
Good free open-source software (or methods) for blacklisting bots
I manage Ruby on Rails servers and need a way to have the loadbalancer reject bots and along with anyone who hits the site more than say, 60 times per minute (minus static file calls). I know CPANEL ...
0
votes
1answer
87 views
Web log file analysis software to measure search crawlers
I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...
12
votes
5answers
833 views
Which bots and spiders should I block in robots.txt?
In order to:
Increase security of my website
Reduce bandwidth requirements
Prevent email address harvesting
1
vote
1answer
240 views
Anti-bot techniques
I want to do some research on anti-bot techniques for my web application. I am aware of Captcha, but want to explore some other techniques. Especially, I want users to visit all the pages before they ...
2
votes
3answers
403 views
How to detect bots programatically
we have a situation where we log visits and visitors on page hits and bots are clogging up our database. We can't use captcha or other techniques like that because this is before we even ask for ...
1
vote
1answer
253 views
How To Suppress Apache Error
I've gone through the trouble of blocking a number of bots that are trying to crawl our site. The issue now is that the following error is taking over the apache error log:
client denied by server ...
1
vote
2answers
985 views
Are there internet users connecting with HTTP/1.0?
Are there internet users connecting with HTTP/1.0? Such as over a proxy. Because I'm going to block HTTP/1.0 since most of the bots are using HTTP/1.0.
0
votes
2answers
116 views
Are there any regularly updated IP databases of Bots?
I am looking for a database of IP Addresses of known bots, so I can identify them in web site traffic.
I have useragent data, but there seem to be plenty that don't give useragents to identify ...
2
votes
1answer
187 views
Are there any regularly updated Bot/Spider/Crawler Databases?
I am looking for a database that is regularly updated of different bots, spiders, and crawlers. I want to be able to identify them in the log files from IIS.
11
votes
8answers
4k views
Why do 'hackers' setup bots repeatedly fill out web forms?
There is something I don't get, one of my web apps has a small form that allows you to enter you name and email address to "subscribe" to a user list for a site I maintain. The site is very low ...
1
vote
5answers
597 views
Is there any way to find out how many hits you receive from bots, on an IIS6 site?
Is there any way to figure out how many bots (even if it's just one type of bot) that hits our site, on a daily, weekly and monthly average?
we have the standard IIS6 web site logs in lots of files.
...
5
votes
6answers
292 views
Monitoring and detecting abusive traffic on webserver
I curious how others monitor and detect crawler robots & IPs that are using up a lot of bandwidth on a webserver? This isn't for preventing DDoS attacks as much as bots run amok. What tools or ...
