Tag Info

Hot answers tagged

5

Robots.txt is a strictly voluntary convention amongst search engines; they're free to ignore it, or implement it in any way they choose. That said, barring the occasional spider looking for email addresses or the like, they pretty much all respect it. Its format and logic are very, very simple, and the default rule is allow (since you can only *dis*allow). ...


5

If your authentication is working there is no reason to hide the url. Id rather focus on that.


3

According to Majestic 12's own page about robots.txt, they fully respect robots exclusion (3rd answer from the top). The robots.txt file is a plain text file in the root of your website, i.e. you place it at: http://www.yourdomain.com/robots.txt and have these lines in the file: User-agent: MJ12bot Disallow: / So if you want to block that bot, I see no ...


3

Looks like Googlebot may be lightly probing your site in search of possible content-duplication issues. Or to see if your site correctly handles non-existent files (by returning a 404 response header) and/or bogus query strings. It may also be testing to see if you may be some kind of link farm if bogus query requests deliver some kind of differing result. ...


3

I'm too young here (reputation) to post all the URLs I need to as hyperlinks, so pardon my parenthesized URLs, please. The forum link from Dan Andreatta, and this other one, have some but not all of what you need. You'll want to use their method of finding the IP numbers, and script something to keep your lists fresh. Then you want something like this, to ...


3

It is not possible for search engines to find it, if it is not linked anywhere (as this is the only way search engines find other information). In order to make sure a search engine does not find it via a robots.txt and have the secrecy, use a double-hash: http://example.com/asdfghjk/12345678 Your robots.txt would disallow anything below asdfghjk: ...


2

Xapian Omega is amazing. And it runs on most operating systems. From the website: Xapian is a highly adaptable toolkit which allows developers to easily add advanced indexing and search facilities to their own applications. It supports the Probabilistic Information Retrieval model and also supports a rich set of boolean query operators. ...


2

You want to throttle the bot, but you don't appear to know WHY you want to do this. Are you experiencing a performance impact? Is the traffic pushing you over a bandwidth or transfer threshold? Throttling a bot "just because" is a waste of effort - If it is not hurting you I suggest that you leave it alone. If it is causing problems you can take steps ...


2

This error can result if there is any issue while importing haystack/backends/whoosh_backend.py. One such example is an import error on from whoosh.spelling import SpellChecker As a result of mixing Whoosh version 2.5.1 with Haystack 1.2.7 (or likely other 1.2.x version of Haystack. The recommended fix for now is to install the older 2.4.1 version of ...


2

I think you want Apache Solr. Lucene is a library, while Solr is a Tomcat application, much more likely what your friend was thinking of.


2

First make sure your DNS has an * record for www pointing at your server IP, this will make sure your site still displays without a www. If you wanted to add the www bit on for consistency of the URL then look at using .htaccess if it's applicable to your situation.


2

Bing FAQ tells about sitemaps and use http://www.sitemaps.org/ protocol.


2

For the op's follow up question: I wonder if they have mentioned all fake bots. Why would someone claim to be majestic12? That would be a false flag operation. Virus disguises itself as a legit bot/process to crawl ip's. The explanation is on the majestic12 faq way down.


2

The purpose of a robots.txt file is to keep crawlers out of certain parts of your website. Not having one should result in all your content being indexed. The implication from the first comment on that Meta question was that the robots.txt file existed but was inaccessible (for whatever reason), rather than not being there at all. That might cause the web ...


1

robots.txt is completely optional. If you have one, standards-compliant crawlers will respect it, if you have none, everything not disallowed in HTML-META elements (Wikipedia) is crawlable.


1

Site will be indexed without limitations. spiders will follow whatever they find. i don't think you want that. some spiders like baidu can be very aggressive about that. it can even evaluate even urls in javascript codes. here is detailed information. http://www.robotstxt.org/orig.html ps. also you will have many 404 logs in your webserver. it's also ...


1

It should be fine as long as your crawling is respecting the robots.txt file of the sites. Searching google for robots.txt will give you a lot of information. Briefly, it is a file for specifying how a crawler/robot accesses the site, allowed and disallowed content, access rate, time of day, etc.


1

There's the google search appliance.. That should do what you're after... although it might cost you an arm, leg and kidney.. http://www.google.co.uk/enterprise/gsa/index.html


1

I haven't had robots.txt on dozens of domains I've had registered, some as far back as 1994, and have never had a problem with them getting placed in google/yahoo, etc. Even my personal website gets 150-200 users a day from google, and doesn't have a robots.txt file. (Love the three minute pause requirement between answering questions. Next I'll get the ...


1

My current solution is this (for NGINX web server): if ($http_user_agent ~* (Yandex) ) { return 444; } This is case insensitive. It returns response 444. This directive looks at the User Agent string and if "Yandex" is detected connection is closed without sending any headers. 444 is a custom error code understood by the Nginx daemon


1

Google generally looks at these pages as different pages, although it's my understanding that Google is working on better assuring whether they are or are not the same content. For now, a re-direct isn't a bad idea. Want to test for yourself? Browse around like one guy did you and you'll see different page ranks on for different trailing slash/no-trailing ...


1

For the past few days Googlebot has been doing the same thing to one of our sites. It appears to be inserting a querystring value that matches a key we use, but expects an integer where Googlebot is supplying a string. (e.g. The parameter should be something like gb=22 but Googlebot is looking for gb=lkcvvzxxz) What's worse, Googlebot is indexing these bad ...


1

According to this forum, the yandex bot is well behaved and respects robots.txt. In particular they say The behaviour of Yandex is quite a lot like that of Google with regard to robots.txt .. The bot doesn't look at the robots.txt every single time it enters the domain. Bots like Yandex, Baudi, and Sohu have all been fairly well behaved ...


1

Are they found along with other Googlebot entries? If not it could be Googlebot is checking links from another website to yours to verify the connection with their algorithms. This means another website has links to your website with those URLs. I don't know if spam or link domains can do something with those URLs or not. As I don't necessarily understand ...


1

Create robots.txt file with the following contents under (/var/www/html): User-agent: YandexBot User-agent: Ezooms Disallow: / Add the following to your httpd.conf file Alias /robots.txt /var/www/html/robots.txt This robots.txt file will now be served for all virtual hosts on your server, overriding any robots.txt file you might have for individual ...


1

I don't think that is provider of any services Google uses. The first result for Google search on that link gives me this page, this seems to be a malicious redirect. http://evuln.com/labs/humbertech.com/ Your site might be infected with some malware, check your .htacces and php files for any redirects. Also I think this would be more suited for ...


1

Check out SearchBlox. Or you could always roll your own permuatation using Lucene, Solr, Nutch and/or other similar tools.


1

Nowhere in the question does it actually state it must be free (does it?) In that case, the suggestion I have is the Google Search Appliance. It seems to support a whole bunch of windows-type stuff, a massive bunch of filetypes, and sharepoint integration. It's probably not cheap, but it is supported, and it's Google. Who knows search better than they ...


1

Like a Google Mini? (didn't think they still had those, but it looks that way)



Only top voted, non community-wiki answers of a minimum length are eligible