Search engines are programs that search documents for specified keywords and returns a list of the documents or websites where the keywords were present.

learn more… | top users | synonyms

4
votes
6answers
7k views

Blocking yandex.ru bot

I want to block all request from yandex.ru search bot. It is very traffic intensive (2GB/day). I first blocked one C class IP range, but it seems this bot appear from different IP ranges. For ...
4
votes
3answers
882 views

Search Engine for Network Shares

We have a clustered storage server that contains many project files like Corel, Indesign, Edius, Adobe Premiere, etc.. Also it has lots of image, audio and video files. We are using sharepoint and ...
3
votes
3answers
144 views

How to prevent discovery of a secure URL?

If I have a url that is used for getting messages and I create it like so: http://www.mydomain.com/somelonghash123456etcetc and this URL allows for other services to POST messages to. Is it possible ...
3
votes
3answers
397 views

Are there any good intranet search engines?

I'm searching for a intranet search engine which is capable of spider our intranet websites and network shares like SMB, NFS and optionaly AFP. Ever better for us would be a search engine which is ...
2
votes
2answers
595 views

How to prevent majestic 12 from indexing a site

We experience a lot of traffic and server load on a web server. All I can find out is majestic12 accessing pages all the time. I wonder how I can prevent majestic12 from indexing the site Do they ...
2
votes
1answer
437 views

whoosh_backend module cannot be found

I recently tried to install haystack with a whoosh search engine. This is to work with django 1.3 on a nginx production server. I've followed the installation instructions for each item (both haystack ...
2
votes
2answers
266 views

Intranet Search engine solution to run on SBS 2008

I'm looking for a web based intranet search solution to index some intranet network shares with PDF / Doc / Textfiles etc. and maybe also an intranet wiki. Microsoft has the Search Server Express, ...
2
votes
1answer
267 views

MediaWiki SearchEngine that behaves like google

searchterm: foo should match foo and foobar searchterm: "foo" should only match foo I tried LuceneSearch and SphinxSearch so far, but I couldn't get any of these to behave like google. foo will only ...
1
vote
6answers
2k views

What happens if a website does not have a robots.txt file?

If the robots.txt file is missing in the root directory of a website, how are things treated as: the site is not indexed at all the site is indexed without any restrictions It should logically be ...
1
vote
3answers
549 views

odd query strings in Googlebot requests

Google's indexing bot (edit: yes, it's Google, IP resolves) seems to be adding arbitrary query strings to our home page. xx.xxx.xx.xxx - - [30/Jun/2009:10:14:37 -0400] "GET /?key=61680 HTTP/1.1" 200 ...
1
vote
2answers
1k views

Search Engine Bot - Large amount of hits

I've started tracking user-agent strings on a website at the start of each session. Looking at the data for this month so far I'm seeing on search engine bot that keeps coming up a lot.. Mozilla/5.0 ...
1
vote
4answers
88 views

search engine (bing) lost my www

I just found that my site in the result of bing was broken, casue bing display wrong domain name: my site domain name: www.mysite.com; bing list my site domain name : mysite.com How can i ask bing ...
1
vote
2answers
161 views

Amazon EC2 hosted services and their effect on local SERPS

I am thinking of hosting a project on Amazon's EC2. However I am a little unclear about possible negative impacts this may have on local search results. The projects main customer base is in the UK ...
1
vote
0answers
154 views

Apache Solr: What is a good strategy for creating a tag/attribute based search for an image

I recently read an article about YayMicro that descries how they used solr to search their photos. I would like doing something similar (but on a smaller scale). I have figured out how to have ...
0
votes
1answer
71 views

Installing Apache Lucene for LAMP server

I have Ubuntu running for LAMP (Linux, Apache, MySQL and PHP) server. To provide better search capability one of my friend recommended to install "Apache Lucene". While reading about it I came to ...
0
votes
1answer
48 views

sitemap for bing?

Google has the sitemap.xml and Yahoo the urllist.txt that you can place in the root of your website to facilitate the indexation process. Does Bing has something similar ?
0
votes
1answer
184 views

Can I use a Google Appliance/Mini to crawl and index sites I don't own?

Maybe this is a stupid question, but... I am working with this company and they said they needed to get "permission" to crawl other people's sites. They have a Google Search Appliance And some ...
0
votes
1answer
105 views

301 redirects and SEO

I have a site which uses the following structure, domain.com/12/43/45/some-post/ When people link to my site some cms's remove the trailing slash and turn it into, domain.com/12/43/45/some-post ...
0
votes
0answers
4 views

Does Google bot(and/or search engines) index a forwarded page? [migrated]

Let say I have foo.bar domain, and I force the user to use the https over http. The question is as browsers just accept and load the forwarded/new page(when the request for http://foo.bar -> ...
0
votes
0answers
7 views

Putting a website in a directory beyond the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution? [migrated]

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
0
votes
1answer
242 views

Requests masking server name / DAPPER-HOST-IP

For the last few days, I've been having a lot of requests to unexisting pages on my server. The worrying part about it is that, when I look at the 404 error log I've built for my site, these requests ...
0
votes
1answer
101 views

prevent search engines indexing depending on domain

We have a dedicated server with a hosting company with a couple of dozens of webs in it. It happens that the nameservers (EG: ns1.domain.com, ns2.domain.com) ip's are coincident with some client webs, ...
0
votes
1answer
159 views

search options based on data

I write an example to be more practive. I need to know if it is possible to next search based on the data I have in my database, fulltextsearch engines in sqlserver-2008 or Apache Solr: Case 1: Data ...
0
votes
1answer
105 views

Web log file analysis software to measure search crawlers

I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...
0
votes
1answer
96 views

/manual/cache folder on my server?

On our site's server, once managed by someone who's no longer with us, there's a folder named "/manual/cache" which contains txt files named+like+this, mostly using pornographic-related keywords. The ...
-1
votes
1answer
211 views

blocking search engine bots directadmin [closed]

I want to block some bad search engine bots like MJ12bot, YandexBot and Ezooms. We have like 200 users in the directadmin environment, and we want to install a "plugin" or "mod" to block those request ...
-2
votes
1answer
83 views

Site redirects to humbertech.com if accessed from search engine [duplicate]

I asked this on stackoverflow but they pointed me here. So this is a copy/paste from that question... I can normally visit my website via url. It all works fine. But when I find it on google the ...