Tagged Questions

The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
12 views

Does the google crawler really guess URL patterns and index pages that were never linked against? [migrated]

I'm experiencing problems with indexed pages which were (probably) never linked to. Here's the setup: Data-Server: Application with RESTful interface which provides the data Website A: Provides the ...
-5
votes
1answer
76 views

Bot being redirected to Google.com when requests Myspace.com… what? [closed]

first time on Serverfault. I'm having a problem connecting to Myspace.com through my server. I've been using mechanize via Python to run a bot, (not spam, crawling for information on musicians) on a ...
0
votes
1answer
32 views

Does a forward web proxy exist that checks and obeys robots.txt on remote domains?

Does there exist a forward proxy server that will lookup and obey robots.txt files on remote internet domains and enforce them on behalf of requesters going via the proxy? e.g. Imagine a website at ...
-1
votes
1answer
95 views

Using wget to get count of pages below a link?

I've been using a sitemapping tool to get a simple count of links below a specific url. The free trial has ended, so I figure that rather than paying $70 for what is very simple functionality, I ...
5
votes
5answers
317 views

How are these 'bad bots' finding my closed webserver?

I've installed Apache a while ago, and a quick look at my access.log shows that all sorts of unknown IPs are connecting, mostly with a status code 403, 404, 400, 408. I have no idea how they're ...
0
votes
0answers
56 views

Methods to prevent malicious crawlers/scrapers and DDoS Attacks

From last couple of weeks I have been experiencing bot attacks on my site. Basically crawlers are running on the site at a high frequency rate resulting in load increase. This results in bandwidth ...
0
votes
1answer
46 views

Any good webcrawler besides DRKSpider

I was having a look at DRKSpider to find problems with a website in our production server, but it seems its export feature generates different outputs (with different content). My goal is to find a ...
0
votes
2answers
40 views

Can access web application from browser but crawler application throws 404 erorr?

I am using an application called Xenu Link Sleuth to try and find broken links on a site we host. When I go to the site through a browser it pops right open. When I try to run it through Xenu it ...
0
votes
1answer
49 views

how to establish connection between drupal and solr

I am working on technologies like drupal & solr.I have installed all required modules.but i need to know that how to crawl data from drupal.& how to form connector between drupal & solr.
0
votes
1answer
53 views

Web log file analysis software to measure search crawlers

I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...
1
vote
1answer
94 views

Yahoo AdCrawler hammering our site

Yahoo AdCrawler is re-trying some URLs repeatedly. The URLs are being given a 302 response code, so I suppose Yahoo should come back and try again "later", but "later" in my book doesn't mean that 7 ...
0
votes
1answer
69 views

crawling [own] webpage for 404s

can you suggest linux tool that can crawl given page in search of 404s? yes - i can use wget in recursive mode or just watch access logs but maybe you have hints for better suited tool? thanks!
0
votes
1answer
148 views

Can I use a Google Appliance/Mini to crawl and index sites I don't own?

Maybe this is a stupid question, but... I am working with this company and they said they needed to get "permission" to crawl other people's sites. They have a Google Search Appliance And some ...
0
votes
3answers
89 views

Google APPS site - not being indexed

Help! Google crawlers appear to be visiting my site but it is not getting indexed, what am I doing wrong? Yahoo has managed to find the mydomain.appspot.com and has indexed successfully (albeit on ...
1
vote
2answers
439 views

Finding all IP ranges blelonging to a specific ISP

I'm having an issue with a certain individual who keeps scraping my site in an aggressive manner; wasting bandwidth and CPU resources. I've already implemented a system which tails my web server ...
1
vote
2answers
290 views

Copy a website and preserve the file & folder structure

I have an old web site running on an ancient version of Oracle Portal that we need to convert to a flat-html structure. Due to damage to the server we are not able to access the administrative ...
2
votes
3answers
137 views

How much HDD space would I need to cache the web while respecting robot.txts?

I want to experiment with creating a web crawler. I'll start with indexing a few medium sized website like Stack Overflow or Smashing Magazine. If it works, I'd like to start crawling the entire web. ...
0
votes
1answer
57 views

Getting web.archive.org to archive website again

I noticed that my website isn't archived anymore by web.archive.org. When I look at http://web.archive.org/web/*/http://www.cnn.com it is clearly visible that it stopped working in july 2008. ...
4
votes
1answer
354 views

Site crawler/spider that tosses results into mysql

It's been suggested that we use mysql for our site's search as it'd be running on the same server that hosts our web server (nginx) and our db (mysql). Since not all of our pages are created from the ...
0
votes
2answers
278 views

Counting the number of pages in a website

What is the easiest way to get a count of the number of pages on a website? I don't want to actually download a local copy the entire site, just get a count of pages on it. Is there a tool (or ...
0
votes
1answer
385 views

Remove subdomain from Google Crawler

I recently removed a sub-domain from my domain so I just have 1 website to manage. However, if I do a google search, my old domain is still there, I removed the sub-domain well over a week ago and if ...
0
votes
1answer
2k views

Sharepoint Crawler is denied access to sites

We create all our site collections programatically with a custom site def/template. Everything works as expected, except for the crawler. It's apparently denied access to the sites. The crawl logs ...
-1
votes
1answer
415 views

Google web crawler cannot find my wordpress posts [closed]

I have a wordpress blog on my own server, which used permanant links containing Chinese characters in urls like http://techblog.zellux.czm.cn/2008/03/ics-lab4-%E7%BB%8F%E9%AA%8C/. Several months ago ...