The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
7 views

Putting a website in a directory beyond the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution? [migrated]

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
2
votes
1answer
633 views

/phpTest/zologize/axa.php? Another botnet?

Staring at the log made me think, what is /phpTest/zologize/axa.php and why are bots looking for it? Previously, I had lots of /HNAP1/ requests. Requesting /HNAP1/ from IPs from log revealed, that all ...
0
votes
1answer
60 views

How to autoban IPs that request URLs with a .php extension?

I've noticed in my linux/apache server logs that bot computers running muieblackcat are periodically trying to access all forms of phpmyadmin.php on my server. It's interesting, because I have a ...
4
votes
1answer
335 views

Why my nginx server receives requests targeting ads.yahoo.com?

I have a nginx proxy facing public internet. A few days ago, I started to receive a torrential amount of requests, targeting URLs that are invalid for my domain. An excerpt from nginx access.log: ...
0
votes
0answers
359 views

Is CSF's recommended Connection Tracking Limit (CT_LIMIT) of 300 connections reasonable?

CSF recommends setting CT_LIMIT to 300 to block small ddos attacks as per http://configserver.com/free/csf/readme.txt I Would like to enable this connection limiting / tracking via CSF, as I have ...
1
vote
1answer
537 views

IIS Request Filtering Rule for User Agent

I'm trying to block requests from a certain bot. I've added a request filtering rule, but I know it is still hitting the site because it shows up in Google Analytics. Here is the filtering rule I ...
0
votes
1answer
244 views

Requests masking server name / DAPPER-HOST-IP

For the last few days, I've been having a lot of requests to unexisting pages on my server. The worrying part about it is that, when I look at the 404 error log I've built for my site, these requests ...
1
vote
3answers
291 views

Chinese bot visiting non-existent page every 5 minutes

Since 02.05.2013 some sort of bot is visiting my website, every 5 minutes. Most of the time it calls this URL, which doesn't exist: /viewtopic.php?f=3&t=849 Always with this user agent ...
0
votes
3answers
156 views

How to handle a DDoS attack on a search engine? [duplicate]

We are running a search engine and are seeing a huge amount of fake search queries coming in over the last days from thousands of IP addresses. There is no real pattern in terms of queries text or IP ...
1
vote
1answer
317 views

How to ban web crawler using fail2ban

I am using nginx and I am always hit by web crawler if I am correct. I tried to configure fail2ban but the IP address cannot be detected by fail2ban. The reason that it is not detected because it ...
0
votes
1answer
81 views

Is there a chance to block images spiders / bots on dedicated servers without using robots.txt or .htaccess?

We know that we can block certain spiders from crawling websites pages using robots.txt or .htaccess or maybe via the Apache configuration File httpd.conf. But that requires to edit may be a large ...
1
vote
1answer
293 views

Configure .htaccess to block bot indexing but allow browser directory listing/viewing

I know in order to show a directory listing of my files in a browser through .htaccess I can use Options +Indexes and to prevent Google and most bots from crawling my directory I can use Options ...
1
vote
3answers
1k views

How to block this URL pattern in Varnish VCL?

My website is getting badly hit by spambots and scrappers, I am using Cloudflare but the problem still remains there. The problem is spambots accessing non-existing urls causing a lot of load to my ...
0
votes
2answers
327 views

Nginx 400 errors due to random encoded string starting with “\x” from random IP addresses

I assume these are some sort of bots, but would like to know what are they trying to do to my server. The logs in questions are below and the IP address has been changed from the original. ...
-4
votes
1answer
93 views

Stop BPimage walker bot [closed]

How can i block all bots except a few including BPImage walker...it is creating a huge amount of load on web site and consuming lots of bandwidth
4
votes
1answer
348 views

Server 2008 adit logs show 1000's of Failure Logon Attempts to the `Admin` account

I was just going through the logs of one of my servers and I noticed there are a ton of failed logon attempts for username Administrator, owner and Admin. I assume that these are bots since there is ...
1
vote
1answer
640 views

How to stop bots using nginx?

I need to detect bots that act as authenticated users and saturate database's available connections by sending repetitive requests. I am wondering whether it is possible to stop dummy them using nginx ...
2
votes
2answers
458 views

Can apache forward/route requests based on http header (loadbalancing and bots)

We have a situation where bots comprise 40% of page requests for a slow-performing web application. Short Term Relief As immediate relief, I thought of doing the following: Setting up another ...
2
votes
2answers
184 views

ethical/legal consideration when redirecting

A webcrawler has bought our site down twice, it ignores our robots.txt and we have had no reply from their customer services or support using both e-mail and twitter. I have had to create url ...
0
votes
1answer
270 views

webcrawler bots load test my website and it fails the test

we run a commerical website with a relavitely few number of customer at any one time ~30 users . Frequently a webcrawler such as google bot, bing bot, and 80legs will bring our site to a grinding ...
2
votes
1answer
444 views

I need to block this BOT or not by a firewall rule?

SCENARIO: Windows Server 2008, IIS 7, SQL Server 2005 I have a company website and i notice in the log this BOT (i think is a BOT) that continually hit some of my pages. ...
4
votes
4answers
701 views

How to prevent bots from trying to guess a link on my site

My logwatch report which I installed recently shows me this: --------------------- httpd Begin ------------------------ 0.78 MB transferred in 5864 responses (1xx 0, 2xx 4900, 3xx 0, 4xx 964, 5xx 0) ...
0
votes
1answer
108 views

Lots of bots making fake calls

I have a small VPS with Apache + php on which i host just a small blog. I've been looking through its logs and seen tons of calls that (i guess) have the hostname faked. Here's a sample. Obviously ...
3
votes
8answers
755 views

Determining if a bot is scraping utility bill content and how to block it

I work for a local government that bills for water usage and garbage collection. I received a call today from a customer that a "New York company" called him asking for his customer number and pin to ...
1
vote
4answers
2k views

Netstat: How to know if it's Human or Bot/Spider/DDOS

I'm using the following command to check how many connections I have from a single ip netstat -anp |grep ':80' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n If it's DDOS attack(500+ ...
8
votes
5answers
6k views

How are these 'bad bots' finding my closed webserver?

I've installed Apache a while ago, and a quick look at my access.log shows that all sorts of unknown IPs are connecting, mostly with a status code 403, 404, 400, 408. I have no idea how they're ...
0
votes
2answers
623 views

Methods to prevent malicious crawlers/scrapers and DDoS Attacks [closed]

From last couple of weeks I have been experiencing bot attacks on my site. Basically crawlers are running on the site at a high frequency rate resulting in load increase. This results in bandwidth ...
1
vote
2answers
1k views

Search Engine Bot - Large amount of hits

I've started tracking user-agent strings on a website at the start of each session. Looking at the data for this month so far I'm seeing on search engine bot that keeps coming up a lot.. Mozilla/5.0 ...
2
votes
2answers
2k views

Blocking by user-agent string in httpd.conf not effective

I'd like to block some spiders and bad bots by user-agent text string for all of my virtual hosts via httpd.conf but have yet to find success. Below are the contents of my http.conf file. Any ideas ...
1
vote
1answer
610 views

Strange request in access.log, how to block?

I am using nginx on my own server, and I noticed a few days ago some strange request in my access.log : 77.50.217.37 - - [19/Aug/2011:17:50:50 +0200] "GET http://images.google.com/ HTTP/1.1" 200 151 ...
0
votes
2answers
286 views

Good free open-source software (or methods) for blacklisting bots

I manage Ruby on Rails servers and need a way to have the loadbalancer reject bots and along with anyone who hits the site more than say, 60 times per minute (minus static file calls). I know CPANEL ...
0
votes
1answer
106 views

Web log file analysis software to measure search crawlers

I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...
13
votes
5answers
1k views

Which bots and spiders should I block in robots.txt?

In order to: Increase security of my website Reduce bandwidth requirements Prevent email address harvesting
1
vote
1answer
298 views

Anti-bot techniques

I want to do some research on anti-bot techniques for my web application. I am aware of Captcha, but want to explore some other techniques. Especially, I want users to visit all the pages before they ...
2
votes
3answers
516 views

How to detect bots programatically

we have a situation where we log visits and visitors on page hits and bots are clogging up our database. We can't use captcha or other techniques like that because this is before we even ask for ...
1
vote
1answer
305 views

How To Suppress Apache Error

I've gone through the trouble of blocking a number of bots that are trying to crawl our site. The issue now is that the following error is taking over the apache error log: client denied by server ...
3
votes
2answers
1k views

Are there internet users connecting with HTTP/1.0?

Are there internet users connecting with HTTP/1.0? Such as over a proxy. Because I'm going to block HTTP/1.0 since most of the bots are using HTTP/1.0.
0
votes
2answers
122 views

Are there any regularly updated IP databases of Bots?

I am looking for a database of IP Addresses of known bots, so I can identify them in web site traffic. I have useragent data, but there seem to be plenty that don't give useragents to identify ...
2
votes
1answer
252 views

Are there any regularly updated Bot/Spider/Crawler Databases?

I am looking for a database that is regularly updated of different bots, spiders, and crawlers. I want to be able to identify them in the log files from IIS.
11
votes
8answers
7k views

Why do 'hackers' setup bots repeatedly fill out web forms?

There is something I don't get, one of my web apps has a small form that allows you to enter you name and email address to "subscribe" to a user list for a site I maintain. The site is very low ...
1
vote
5answers
625 views

Is there any way to find out how many hits you receive from bots, on an IIS6 site?

Is there any way to figure out how many bots (even if it's just one type of bot) that hits our site, on a daily, weekly and monthly average? we have the standard IIS6 web site logs in lots of files. ...
5
votes
6answers
350 views

Monitoring and detecting abusive traffic on webserver [closed]

I curious how others monitor and detect crawler robots & IPs that are using up a lot of bandwidth on a webserver? This isn't for preventing DDoS attacks as much as bots run amok. What tools or ...