A web-crawler (also known as a web-spider) traverses the webpages of the internet by following the links of urls contained within each webpage. There is usually an initial seed of URLs from which the crawler is given to initialize its crawl.

learn more… | top users | synonyms

1
vote
3answers
3k views

Blocking 'good' bots in nginx with multiple conditions for certain off-limits URL's where humans can go

After 2 days of searching/trying/failing I decided to post this here, I haven't found any example of someone doing the same nor what I tried seems to be working OK. I'm trying to send a 403 to bots ...
0
votes
0answers
22 views

Having concurrent Nutch Crawls-Solr Indexing failing

We have two jobs Nutch Crawling then indexing-. These jobs uses same nutch library and binaries directories . An example of the directory structure we have in Ubuntu server is shown below. ...
22
votes
4answers
6k views

Does a company have implied right to crawl my website?

I have found out that McAfee SiteAdvisor has reported my website as "may be having security issues". I care little about whatever McAfee thinks of my website (I can secure it myself and if not, ...
-9
votes
2answers
69 views

Get all urls of a website [closed]

I wanna build a tool which scans a website for all urls, but not the urls within the page but of the site self, but I don't know how. Could anyone give me an example how I can start? Example: ...
0
votes
1answer
100 views

Tool or website to check links

We supply Magento and Typo3 installations to customers. To improve QA we wanted to use an automatic link checker to check for broken and/or outdated links - automatically. We want to check all links ...
0
votes
1answer
92 views

My website might have problems being indexed by Google bots?

http://ptcsavjetovaliste.org and I think because it is in Croatian language it might have problems being indexed because of letters like čćžšđ?! Look at the crawler errors I see in Webmaster tools... ...
3
votes
1answer
94 views

Why is googlebot requesting robots.txt from my SSH server?

I run ossec on my server and periodically I receive a warning like this: Received From: myserver->/var/log/auth.log Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version ...
-3
votes
1answer
68 views

is there any web crawler can get access to users region for download? [closed]

Actually I am using Httrack as a web crawler, can it use my credentials to access members area and download the zip files because they are restricted from public access. Thank you in advance. ...
1
vote
1answer
165 views

Yahoo AdCrawler hammering our site

Yahoo AdCrawler is re-trying some URLs repeatedly. The URLs are being given a 302 response code, so I suppose Yahoo should come back and try again "later", but "later" in my book doesn't mean that 7 ...
0
votes
2answers
221 views

Protection against scrapping with nginx

This morning we had a crawler going nuts on our server hitting our site almost 100 times per second. We'd like to add a protection for this. I guess I'' have to use HttpLimitReqModule but I don't ...
0
votes
1answer
38 views

Referrer in access.log is a directory

It seems that the referrer on the following log is a folder. 112.200.208.5 - - [29/Jul/2013:20:43:14 +0800] "GET /sites/default/files/download/argie/pos-code.zip HTTP/1.1" 206 294677 ...
0
votes
1answer
149 views

How to Block Web Crawler from Downloading File

Is it possible to block web crawler from downloading files (like zip file) in my server? I supposed to create a PHP script using cookies to track visitors specially web crawlers to login/register ...
1
vote
0answers
191 views

How to Exclude Log Using Fail2Ban logpath with Wildcard Settings

I'm using wildcard in the logpath value as shown below: [http-get-dos] enabled = true filter = http-get-dos logpath = /var/log/ispconfig/httpd/*/access.log maxretry = 250 findtime = 300 #ban for 10 ...
1
vote
1answer
339 views

How to ban web crawler using fail2ban

I am using nginx and I am always hit by web crawler if I am correct. I tried to configure fail2ban but the IP address cannot be detected by fail2ban. The reason that it is not detected because it ...
3
votes
2answers
371 views

Does google's web crawler download binary files?

My Google-fu is failing me right now. I'm trying to figure out whether Google's web crawler downloads non-image binary files when it spiders sites. I know it downloads (and indexes) images and PDFs, ...
0
votes
1answer
86 views

Is there a chance to block images spiders / bots on dedicated servers without using robots.txt or .htaccess?

We know that we can block certain spiders from crawling websites pages using robots.txt or .htaccess or maybe via the Apache configuration File httpd.conf. But that requires to edit may be a large ...
2
votes
2answers
62 views

Strange “GET /api/levels/ ” and “GET /play/” requests in logs

I've setup new Amazon EC2 instance. In a day or two started to get strange "GET" requests from the "google bot-like" IP's (eg 66.249.76.84, 66.249.74.152) about one in 10 seconds (some examples): ...
-2
votes
1answer
22 views

How to fix the correct html file as homepage? [closed]

A search for my domain shows defaulthomepage along with html file that my hosting service, using plesk 9.5.4, provides as a placeholder. I have since deleted the old index file and added a new ...
2
votes
4answers
4k views

Finding all IP ranges belonging to a specific ISP

I'm having an issue with a certain individual who keeps scraping my site in an aggressive manner; wasting bandwidth and CPU resources. I've already implemented a system which tails my web server ...
0
votes
1answer
96 views

How do I scan my folders for a website? Like a crawler?

I'd like to scan all the url's on my website as well as get the files in them, but the thing is, there are too many for me to do this manually so how would I do this? I'd like it formatted anyway as ...
1
vote
1answer
118 views

Googlebot repeatedly looks for files that aren't on my server

I'm hosting a site for a volunteer organization. I've moved the site to WordPress, but it wasn't always that way. I suspect at one point it was hacked badly. My Apache error log file has grown to ...
1
vote
2answers
605 views

How to create a global robots.txt that gets appended to each domain's own robots.txt on Apache?

I know can create ONE robots.txt file for all domains on an Apache server*, but I want to append to each domain's (if pre-existing) robots.txt. I want some general rules in place for all domains, but ...
1
vote
3answers
1k views

How to block this URL pattern in Varnish VCL?

My website is getting badly hit by spambots and scrappers, I am using Cloudflare but the problem still remains there. The problem is spambots accessing non-existing urls causing a lot of load to my ...
0
votes
2answers
642 views

Methods to prevent malicious crawlers/scrapers and DDoS Attacks [closed]

From last couple of weeks I have been experiencing bot attacks on my site. Basically crawlers are running on the site at a high frequency rate resulting in load increase. This results in bandwidth ...
2
votes
3answers
259 views

How much HDD space would I need to cache the web while respecting robot.txts? [closed]

I want to experiment with creating a web crawler. I'll start with indexing a few medium sized website like Stack Overflow or Smashing Magazine. If it works, I'd like to start crawling the entire web. ...
5
votes
3answers
2k views

How do I use robots.txt to disallow crawling for only my subdomains?

If I want my main website to on search engines, but none of the subdomains to be, should I just put the "disallow all" robots.txt in the directories of the subdomains? If I do, will my main domain ...
0
votes
1answer
59 views

Webmaster randomly reporting a massive increase in 404s (apparently from old sitemaps)

Well, I'm stumped. Several months back, we launched a totally new website, replacing a legacy system that was pretty messy. Part of the mess was many, many pages created that really didn't need to be ...
2
votes
2answers
188 views

ethical/legal consideration when redirecting

A webcrawler has bought our site down twice, it ignores our robots.txt and we have had no reply from their customer services or support using both e-mail and twitter. I have had to create url ...
4
votes
3answers
936 views

Search Engine for Network Shares

We have a clustered storage server that contains many project files like Corel, Indesign, Edius, Adobe Premiere, etc.. Also it has lots of image, audio and video files. We are using sharepoint and ...
0
votes
1answer
780 views

What is “/admin/Y-ivrrecording.php?php=info&ip=uname”?

Help me please, I found an IP address from Korea try to do something with my web server by put this '/admin/Y-ivrrecording.php?php=info&ip=uname' like search a filename from my web server. I ...
4
votes
3answers
205 views

How often does Google's web spiders crawl the web?

Just a few hours after having made some changes in the HTML of my site, I found that Google had updated its search result against my website. The Internet is so huge, how did the Google crawler do ...
-4
votes
1answer
536 views

Bot being redirected to Google.com when requests Myspace.com… what? [closed]

first time on Serverfault. I'm having a problem connecting to Myspace.com through my server. I've been using mechanize via Python to run a bot, (not spam, crawling for information on musicians) on a ...
1
vote
1answer
228 views

Does a forward web proxy exist that checks and obeys robots.txt on remote domains?

Does there exist a forward proxy server that will lookup and obey robots.txt files on remote internet domains and enforce them on behalf of requesters going via the proxy? e.g. Imagine a website at ...
-2
votes
1answer
188 views

Whats the best server-language for programing a Webcrawler? [closed]

I would like to ask what language: ASP.NET / Ruby / CGI / Perl / Python / ColdFusion... would be the bes for programming a Webcrawler and for processing the contained information???? (it should ...
-1
votes
1answer
739 views

Using wget to get count of pages below a link?

I've been using a sitemapping tool to get a simple count of links below a specific url. The free trial has ended, so I figure that rather than paying $70 for what is very simple functionality, I ...
8
votes
5answers
6k views

How are these 'bad bots' finding my closed webserver?

I've installed Apache a while ago, and a quick look at my access.log shows that all sorts of unknown IPs are connecting, mostly with a status code 403, 404, 400, 408. I have no idea how they're ...
0
votes
2answers
1k views

Is it good idea to ban amazonaws.com [closed]

Site are crawled by anonymous bot hosted on amazon ec2. This robot doesn't respect robots.txt and creates high load on web server so I added check if reverse IP for request ends with "amazonaws.com" ...
4
votes
4answers
968 views

How i can run nutch on linux kernel?

I want to run nutch on the linux kernel,I have loged in as a root user, I have setted all the environment variable and nutch file setting. I have created a url.txt file which content the url to ...
0
votes
1answer
218 views

Any good webcrawler besides DRKSpider

I was having a look at DRKSpider to find problems with a website in our production server, but it seems its export feature generates different outputs (with different content). My goal is to find a ...
1
vote
1answer
635 views

Strange request in access.log, how to block?

I am using nginx on my own server, and I noticed a few days ago some strange request in my access.log : 77.50.217.37 - - [19/Aug/2011:17:50:50 +0200] "GET http://images.google.com/ HTTP/1.1" 200 151 ...
0
votes
3answers
163 views

Open Source Crawler

I came across an open source crawler that recently hit my site. I was wondering, 1. How do you get a list of sites to crawl? 2. Can you get a list of sites to crawl in your city? 3. If you have ...
2
votes
2answers
2k views

Apache crashing with memory/cpu overload when google crawler visits site

I have a site with low traffic, less than 500 hits a day. It has a 6G of memory and it is way underutilized, on average 5% is in use. But as soon as googlebot establishes a connection to my ...
0
votes
1answer
187 views

Can I use a Google Appliance/Mini to crawl and index sites I don't own?

Maybe this is a stupid question, but... I am working with this company and they said they needed to get "permission" to crawl other people's sites. They have a Google Search Appliance And some ...
0
votes
3answers
129 views

Google APPS site - not being indexed

Help! Google crawlers appear to be visiting my site but it is not getting indexed, what am I doing wrong? Yahoo has managed to find the mydomain.appspot.com and has indexed successfully (albeit on ...
0
votes
2answers
70 views

Can access web application from browser but crawler application throws 404 erorr?

I am using an application called Xenu Link Sleuth to try and find broken links on a site we host. When I go to the site through a browser it pops right open. When I try to run it through Xenu it ...
0
votes
1answer
109 views

Web log file analysis software to measure search crawlers

I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...
0
votes
1answer
72 views

how to establish connection between drupal and solr

I am working on technologies like drupal & solr.I have installed all required modules.but i need to know that how to crawl data from drupal.& how to form connector between drupal & solr.
7
votes
8answers
3k views

How can I tell how often Google crawls my site?

I've started a relatively new website, and I submitted it to google and everything. I use google's webmaster tools as well. I'm wondering how to figure out the frequency of google's spider accessing ...
1
vote
1answer
919 views

How big would a MySQL database be if I save all webpages' title and URL in it? [closed]

For learning purposes, I want to make a simple web indexer which crawls the web and saves all found pages in a MySQL database with their titles and URLs, with this table (the page's content is not ...
1
vote
3answers
380 views

Should I ban spiders?

A rails template script that I've been looking at automatically adds User-Agent: and Dissalow: in robots.txt thereby banning all spiders from the site What are the benefits of banning spiders and why ...