Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

If I have a url that is used for getting messages and I create it like so: http://www.mydomain.com/somelonghash123456etcetc and this URL allows for other services to POST messages to. Is it possible for a search engine robot to find it? I don't want to have it in my robots.txt because that will expose it already to anybody that sees the robots file.

Of course I will put in other authentication in the app, but not having anyone discovering that URL is the first step.

Any common methods?

share|improve this question
add comment

3 Answers 3

up vote 3 down vote accepted

It is not possible for search engines to find it, if it is not linked anywhere (as this is the only way search engines find other information). In order to make sure a search engine does not find it via a robots.txt and have the secrecy, use a double-hash:

http://example.com/asdfghjk/12345678

Your robots.txt would disallow anything below asdfghjk:

User-agent: *
Disallow: /asdfghjk/

But anyone that does not know the full path will still not be able to get the second url part from looking at it.

share|improve this answer
2  
Strictly speaking robots.txt doesn't disallow anything. It's merely a suggestion to web crawlers as to what they should and should not index. And more than a few of them ignore those suggestions. Using Robots.txt and "hiding" URLs in the way you suggest does not actually secure anything. –  RobM Oct 7 '11 at 10:15
1  
you are correct. However, there are use cases in which you don't want to have "security by obscurity" instead of real security, and from the question I deduce, that this is the case (e.g. by having application keys for getting a rss-feed without authentication but still with some privacy). Also, search-engines still only find something by seeing a link or by deducing certain urls, so a 32-char-string should provide a sufficient protection against getting find. –  Lars Oct 7 '11 at 10:18
add comment

If your authentication is working there is no reason to hide the url. Id rather focus on that.

share|improve this answer
add comment

you could put in something identifiable such as /secretpage[hash]123456etcetc then grep the apache logs which an automated script that block access if one ip tries to access a secretpage url too often.

In reality though, miademora is correct. Get your authentication secure (2 factor?) and ditch the security by obscurity.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.