Search engines are programs that search documents for specified keywords and returns a list of the documents or websites where the keywords were present.

learn more… | top users | synonyms

0
votes
0answers
34 views

Having concurrent Nutch Crawls-Solr Indexing failing

We have two jobs Nutch Crawling then indexing-. These jobs uses same nutch library and binaries directories . An example of the directory structure we have in Ubuntu server is shown below. ...
0
votes
1answer
254 views

Requests masking server name / DAPPER-HOST-IP

For the last few days, I've been having a lot of requests to unexisting pages on my server. The worrying part about it is that, when I look at the 404 error log I've built for my site, these requests ...
-2
votes
1answer
91 views

Site redirects to humbertech.com if accessed from search engine [duplicate]

I asked this on stackoverflow but they pointed me here. So this is a copy/paste from that question... I can normally visit my website via url. It all works fine. But when I find it on google the ...
-1
votes
1answer
229 views

blocking search engine bots directadmin [closed]

I want to block some bad search engine bots like MJ12bot, YandexBot and Ezooms. We have like 200 users in the directadmin environment, and we want to install a "plugin" or "mod" to block those request ...
0
votes
1answer
71 views

Installing Apache Lucene for LAMP server

I have Ubuntu running for LAMP (Linux, Apache, MySQL and PHP) server. To provide better search capability one of my friend recommended to install "Apache Lucene". While reading about it I came to ...
0
votes
1answer
107 views

prevent search engines indexing depending on domain

We have a dedicated server with a hosting company with a couple of dozens of webs in it. It happens that the nameservers (EG: ns1.domain.com, ns2.domain.com) ip's are coincident with some client webs, ...
4
votes
3answers
965 views

Search Engine for Network Shares

We have a clustered storage server that contains many project files like Corel, Indesign, Edius, Adobe Premiere, etc.. Also it has lots of image, audio and video files. We are using sharepoint and ...
2
votes
1answer
471 views

whoosh_backend module cannot be found

I recently tried to install haystack with a whoosh search engine. This is to work with django 1.3 on a nginx production server. I've followed the installation instructions for each item (both haystack ...
1
vote
2answers
162 views

Amazon EC2 hosted services and their effect on local SERPS

I am thinking of hosting a project on Amazon's EC2. However I am a little unclear about possible negative impacts this may have on local search results. The projects main customer base is in the UK ...
3
votes
3answers
145 views

How to prevent discovery of a secure URL?

If I have a url that is used for getting messages and I create it like so: http://www.mydomain.com/somelonghash123456etcetc and this URL allows for other services to POST messages to. Is it possible ...
2
votes
2answers
270 views

Intranet Search engine solution to run on SBS 2008

I'm looking for a web based intranet search solution to index some intranet network shares with PDF / Doc / Textfiles etc. and maybe also an intranet wiki. Microsoft has the Search Server Express, ...
1
vote
2answers
1k views

Search Engine Bot - Large amount of hits

I've started tracking user-agent strings on a website at the start of each session. Looking at the data for this month so far I'm seeing on search engine bot that keeps coming up a lot.. Mozilla/5.0 ...
1
vote
0answers
154 views

Apache Solr: What is a good strategy for creating a tag/attribute based search for an image

I recently read an article about YayMicro that descries how they used solr to search their photos. I would like doing something similar (but on a smaller scale). I have figured out how to have ...
2
votes
1answer
271 views

MediaWiki SearchEngine that behaves like google

searchterm: foo should match foo and foobar searchterm: "foo" should only match foo I tried LuceneSearch and SphinxSearch so far, but I couldn't get any of these to behave like google. foo will only ...
0
votes
1answer
159 views

search options based on data

I write an example to be more practive. I need to know if it is possible to next search based on the data I have in my database, fulltextsearch engines in sqlserver-2008 or Apache Solr: Case 1: Data ...
3
votes
3answers
403 views

Are there any good intranet search engines?

I'm searching for a intranet search engine which is capable of spider our intranet websites and network shares like SMB, NFS and optionaly AFP. Ever better for us would be a search engine which is ...
0
votes
1answer
111 views

Web log file analysis software to measure search crawlers

I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...
0
votes
1answer
188 views

Can I use a Google Appliance/Mini to crawl and index sites I don't own?

Maybe this is a stupid question, but... I am working with this company and they said they needed to get "permission" to crawl other people's sites. They have a Google Search Appliance And some ...
1
vote
6answers
2k views

What happens if a website does not have a robots.txt file?

If the robots.txt file is missing in the root directory of a website, how are things treated as: the site is not indexed at all the site is indexed without any restrictions It should logically be ...
0
votes
1answer
96 views

/manual/cache folder on my server?

On our site's server, once managed by someone who's no longer with us, there's a folder named "/manual/cache" which contains txt files named+like+this, mostly using pornographic-related keywords. The ...
2
votes
2answers
677 views

How to prevent majestic 12 from indexing a site

We experience a lot of traffic and server load on a web server. All I can find out is majestic12 accessing pages all the time. I wonder how I can prevent majestic12 from indexing the site Do they ...
4
votes
6answers
7k views

Blocking yandex.ru bot

I want to block all request from yandex.ru search bot. It is very traffic intensive (2GB/day). I first blocked one C class IP range, but it seems this bot appear from different IP ranges. For ...
1
vote
4answers
88 views

search engine (bing) lost my www

I just found that my site in the result of bing was broken, casue bing display wrong domain name: my site domain name: www.mysite.com; bing list my site domain name : mysite.com How can i ask bing ...
0
votes
1answer
107 views

301 redirects and SEO

I have a site which uses the following structure, domain.com/12/43/45/some-post/ When people link to my site some cms's remove the trailing slash and turn it into, domain.com/12/43/45/some-post ...
0
votes
1answer
48 views

sitemap for bing?

Google has the sitemap.xml and Yahoo the urllist.txt that you can place in the root of your website to facilitate the indexation process. Does Bing has something similar ?
1
vote
3answers
562 views

odd query strings in Googlebot requests

Google's indexing bot (edit: yes, it's Google, IP resolves) seems to be adding arbitrary query strings to our home page. xx.xxx.xx.xxx - - [30/Jun/2009:10:14:37 -0400] "GET /?key=61680 HTTP/1.1" 200 ...