A web-crawler (also known as a web-spider) traverses the webpages of the internet by following the links of urls contained within each webpage. There is usually an initial seed of URLs from which the crawler is given to initialize its crawl.

learn more… | top users | synonyms

1
vote
3answers
2k views

Blocking 'good' bots in nginx with multiple conditions for certain off-limits URL's where humans can go

After 2 days of searching/trying/failing I decided to post this here, I haven't found any example of someone doing the same nor what I tried seems to be working OK. I'm trying to send a 403 to bots ...
0
votes
3answers
125 views

Google APPS site - not being indexed

Help! Google crawlers appear to be visiting my site but it is not getting indexed, what am I doing wrong? Yahoo has managed to find the mydomain.appspot.com and has indexed successfully (albeit on ...
0
votes
2answers
140 views

Protection against scrapping with nginx

This morning we had a crawler going nuts on our server hitting our site almost 100 times per second. We'd like to add a protection for this. I guess I'' have to use HttpLimitReqModule but I don't ...
0
votes
1answer
62 views

Tool or website to check links

We supply Magento and Typo3 installations to customers. To improve QA we wanted to use an automatic link checker to check for broken and/or outdated links - automatically. We want to check all links ...
0
votes
1answer
113 views

How to Block Web Crawler from Downloading File

Is it possible to block web crawler from downloading files (like zip file) in my server? I supposed to create a PHP script using cookies to track visitors specially web crawlers to login/register ...
0
votes
1answer
57 views

Webmaster randomly reporting a massive increase in 404s (apparently from old sitemaps)

Well, I'm stumped. Several months back, we launched a totally new website, replacing a legacy system that was pretty messy. Part of the mess was many, many pages created that really didn't need to be ...
0
votes
1answer
69 views

how to establish connection between drupal and solr

I am working on technologies like drupal & solr.I have installed all required modules.but i need to know that how to crawl data from drupal.& how to form connector between drupal & solr.
1
vote
0answers
87 views

How to Exclude Log Using Fail2Ban logpath with Wildcard Settings

I'm using wildcard in the logpath value as shown below: [http-get-dos] enabled = true filter = http-get-dos logpath = /var/log/ispconfig/httpd/*/access.log maxretry = 250 findtime = 300 #ban for 10 ...
0
votes
0answers
86 views

Sticky sessions with ELB except for crawlers (or at least google)?

I have a cluster of servers behind an elastic load ballancer. We need to use sticky sessions for users, but for Google crawler we would like it to be non sticky. Is this possible?