Hot answers tagged

6

According to Majestic 12's own page about robots.txt, they fully respect robots exclusion (3rd answer from the top). The robots.txt file is a plain text file in the root of your website, i.e. you place it at: http://www.yourdomain.com/robots.txt and have these lines in the file: User-agent: MJ12bot Disallow: / So if you want to block that bot, I see no ...


5

Robots.txt is a strictly voluntary convention amongst search engines; they're free to ignore it, or implement it in any way they choose. That said, barring the occasional spider looking for email addresses or the like, they pretty much all respect it. Its format and logic are very, very simple, and the default rule is allow (since you can only *dis*allow). ...


5

If your authentication is working there is no reason to hide the url. Id rather focus on that.


4

Looks like Googlebot may be lightly probing your site in search of possible content-duplication issues. Or to see if your site correctly handles non-existent files (by returning a 404 response header) and/or bogus query strings. It may also be testing to see if you may be some kind of link farm if bogus query requests deliver some kind of differing result. ...


4

It is not possible for search engines to find it, if it is not linked anywhere (as this is the only way search engines find other information). In order to make sure a search engine does not find it via a robots.txt and have the secrecy, use a double-hash: http://example.com/asdfghjk/12345678 Your robots.txt would disallow anything below asdfghjk: User-...


3

For the op's follow up question: I wonder if they have mentioned all fake bots. Why would someone claim to be majestic12? That would be a false flag operation. Virus disguises itself as a legit bot/process to crawl ip's. The explanation is on the majestic12 faq way down.


3

The purpose of a robots.txt file is to keep crawlers out of certain parts of your website. Not having one should result in all your content being indexed. The implication from the first comment on that Meta question was that the robots.txt file existed but was inaccessible (for whatever reason), rather than not being there at all. That might cause the web ...


3

I'm too young here (reputation) to post all the URLs I need to as hyperlinks, so pardon my parenthesized URLs, please. The forum link from Dan Andreatta, and this other one, have some but not all of what you need. You'll want to use their method of finding the IP numbers, and script something to keep your lists fresh. Then you want something like this, to ...


2

Bing FAQ tells about sitemaps and use http://www.sitemaps.org/ protocol.


2

First make sure your DNS has an * record for www pointing at your server IP, this will make sure your site still displays without a www. If you wanted to add the www bit on for consistency of the URL then look at using .htaccess if it's applicable to your situation.


2

Xapian Omega is amazing. And it runs on most operating systems. From the website: Xapian is a highly adaptable toolkit which allows developers to easily add advanced indexing and search facilities to their own applications. It supports the Probabilistic Information Retrieval model and also supports a rich set of boolean query operators. ...


2

You want to throttle the bot, but you don't appear to know WHY you want to do this. Are you experiencing a performance impact? Is the traffic pushing you over a bandwidth or transfer threshold? Throttling a bot "just because" is a waste of effort - If it is not hurting you I suggest that you leave it alone. If it is causing problems you can take steps ...


2

I think you want Apache Solr. Lucene is a library, while Solr is a Tomcat application, much more likely what your friend was thinking of.


2

This error can result if there is any issue while importing haystack/backends/whoosh_backend.py. One such example is an import error on from whoosh.spelling import SpellChecker As a result of mixing Whoosh version 2.5.1 with Haystack 1.2.7 (or likely other 1.2.x version of Haystack. The recommended fix for now is to install the older 2.4.1 version of ...


1

This is a really hard one to answer, and certainly not something you'll get a clear answer from google or amazon on. We've gone through the same routes/questions and we're a major SEO player in the UK with Google. The long and short is that no-one really seems to know or can be sure, some people say yes others say no. If SEO and search results is a big ...


1

Like a Google Mini? (didn't think they still had those, but it looks that way)


1

Well, the regex in your RewriteCond demands that the User Agent start with bingbot. That's what the ^ in the regex does. ^bingbot/.* Since the User Agent (from your log example) doesn't start with that, it won't match and skips the Rule. "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" Remove the ^ and it should work, though I'...


1

Site will be indexed without limitations. spiders will follow whatever they find. i don't think you want that. some spiders like baidu can be very aggressive about that. it can even evaluate even urls in javascript codes. here is detailed information. http://www.robotstxt.org/orig.html ps. also you will have many 404 logs in your webserver. it's also ...


1

robots.txt is completely optional. If you have one, standards-compliant crawlers will respect it, if you have none, everything not disallowed in HTML-META elements (Wikipedia) is crawlable.


1

There's the google search appliance.. That should do what you're after... although it might cost you an arm, leg and kidney.. http://www.google.co.uk/enterprise/gsa/index.html


1

This blog post is one of the better resources I've come across. It provides a great overview, downloadable ADMX and ADML files, as well as what's required to properly implement them through a GPO.


1

It should be fine as long as your crawling is respecting the robots.txt file of the sites. Searching google for robots.txt will give you a lot of information. Briefly, it is a file for specifying how a crawler/robot accesses the site, allowed and disallowed content, access rate, time of day, etc.


1

(I could not find a way to add a comment but) Also, I would like to add that not having a robots.txt is also a problem in the sense that you will not be able to provide a Sitemap for it. Remember that Sitemap's are only located by either them being specified in the Robots.txt file or through direct submission to search engines, but of course the latter means ...


1

I haven't had robots.txt on dozens of domains I've had registered, some as far back as 1994, and have never had a problem with them getting placed in google/yahoo, etc. Even my personal website gets 150-200 users a day from google, and doesn't have a robots.txt file. (Love the three minute pause requirement between answering questions. Next I'll get the ...


1

My current solution is this (for NGINX web server): if ($http_user_agent ~* (Yandex) ) { return 444; } This is case insensitive. It returns response 444. This directive looks at the User Agent string and if "Yandex" is detected connection is closed without sending any headers. 444 is a custom error code understood by the Nginx daemon


1

According to this forum, the yandex bot is well behaved and respects robots.txt. In particular they say The behaviour of Yandex is quite a lot like that of Google with regard to robots.txt .. The bot doesn't look at the robots.txt every single time it enters the domain. Bots like Yandex, Baudi, and Sohu have all been fairly well behaved ...


1

Google generally looks at these pages as different pages, although it's my understanding that Google is working on better assuring whether they are or are not the same content. For now, a re-direct isn't a bad idea. Want to test for yourself? Browse around like one guy did you and you'll see different page ranks on for different trailing slash/no-trailing ...


1

For the past few days Googlebot has been doing the same thing to one of our sites. It appears to be inserting a querystring value that matches a key we use, but expects an integer where Googlebot is supplying a string. (e.g. The parameter should be something like gb=22 but Googlebot is looking for gb=lkcvvzxxz) What's worse, Googlebot is indexing these bad ...



Only top voted, non community-wiki answers of a minimum length are eligible