A web-crawler (also known as a web-spider) traverses the webpages of the internet by following the links of urls contained within each webpage. There is usually an initial seed of URLs from which the crawler is given to initialize its crawl.

learn more… | top users | synonyms

0
votes
3answers
133 views

Google APPS site - not being indexed

Help! Google crawlers appear to be visiting my site but it is not getting indexed, what am I doing wrong? Yahoo has managed to find the mydomain.appspot.com and has indexed successfully (albeit on ...
0
votes
2answers
322 views

Protection against scrapping with nginx

This morning we had a crawler going nuts on our server hitting our site almost 100 times per second. We'd like to add a protection for this. I guess I'' have to use HttpLimitReqModule but I don't ...
0
votes
1answer
126 views

Switching between multiple authentication types on same URL

I have a secure SSO site that uses Shibboleth authentication and SAML identity provider. I need to allow a Google Search Appliance crawler to come index the URL's. I have a requirement to change on ...
0
votes
1answer
155 views

Tool or website to check links

We supply Magento and Typo3 installations to customers. To improve QA we wanted to use an automatic link checker to check for broken and/or outdated links - automatically. We want to check all links ...
0
votes
1answer
261 views

How to Block Web Crawler from Downloading File

Is it possible to block web crawler from downloading files (like zip file) in my server? I supposed to create a PHP script using cookies to track visitors specially web crawlers to login/register ...
0
votes
1answer
59 views

Webmaster randomly reporting a massive increase in 404s (apparently from old sitemaps)

Well, I'm stumped. Several months back, we launched a totally new website, replacing a legacy system that was pretty messy. Part of the mess was many, many pages created that really didn't need to be ...
0
votes
1answer
75 views

how to establish connection between drupal and solr

I am working on technologies like drupal & solr.I have installed all required modules.but i need to know that how to crawl data from drupal.& how to form connector between drupal & solr.
1
vote
0answers
54 views

What are the symptoms of an overloaded webserver

I'm maintaining some web crawlers. I want to improve our load/throttling system to be more intelligent. Of cause I look at response codes, and throttle up or down based on that. I would though like ...
1
vote
0answers
309 views

How to Exclude Log Using Fail2Ban logpath with Wildcard Settings

I'm using wildcard in the logpath value as shown below: [http-get-dos] enabled = true filter = http-get-dos logpath = /var/log/ispconfig/httpd/*/access.log maxretry = 250 findtime = 300 #ban for 10 ...
0
votes
0answers
36 views

How to make Google crawl my site using IPv6 address when my domain name has both IPv4 and IPv6 addresses?

My domain name has both IPv4 and IPv6 addresses assigned. IPv4 connection to Google can't be available all the time due to restrictions of my campus network, but IPv6 is available all the time. ...
0
votes
0answers
89 views

Having concurrent Nutch Crawls-Solr Indexing failing

We have two jobs Nutch Crawling then indexing-. These jobs uses same nutch library and binaries directories . An example of the directory structure we have in Ubuntu server is shown below. ...