6

I have a URL that I would like to share with my friend who needs the information on it, but I do not want the public to see it.

The URL refers to an index.html, which is in my public_html and readable for everyone, say:

public_html/secret23/index.html

Nothing links to that URL `.../~dehmann/secret23'. Can I trust that search engines will not find and index it? So no one except my friend will be able to see it? (Assume that my friend will never leak the URL.)

I think the answer is yes, since Google also uses this technique in Picasa and Google Documents, to publish documents, where the user doesn't want search engines to see it. But I just want to make sure this is how it works (before my sensitive information is out there for everyone to see).

UPDATE: To clarify, my question boils down to: Is it true that Google (or, anyone) can not recursively list the content of my public_html? Or can they?

Improve this question
2
  • In reply to your update. If directory listing is turned off and assuming no hack is used, the answer is: NO, they cannot list your directory. But information has a funny way of getting out.
    – Donald Byrd
    Commented Oct 12, 2009 at 18:41
  • Can you please list the actual URL here?
    – Brian Webster
    Commented Jun 9, 2010 at 22:06

11 Answers 11

Reset to default
9

Nobody's mentioned logging yet. If you are using plain HTTP, then any proxy / firewall device in between your server and your end user can log your URL and administrators of those devices will be able to see the "supposedly private" URL being accessed.

If you want to be sure your URL won't be leaked even to intermediate devices like proxies and firewalls, then you ought to protect your resources with SSL. Of course, you still need to trust the end-user's PC in that scenario. But, assuming you trust their PC, and you trust your own server, then you need to use SSL to make sure nobody in between can sniff the URL, either off the network or from a proxy / firewall log file.

IMHO, if you use SSL, then your URL could be as secure as a username/password combination, since in both cases the "secret" is transmitted encrypted. The drawback with a URL as opposed to username/password, though, is the risk that your users accidentally type in the URL without the https:// prefix ... the minute they type in http://, then the secret is out. Username/password wouldn't have that problem since your server could be configured to not challenge for username/password on an insecure connection, and instead redirect first.

Improve this answer
1
  • 1
    +1 Logging is a good point. Allowing usage logs to become public is one way URLs get leaked.
    – Donald Byrd
    Commented Oct 12, 2009 at 12:41
9

If you're mostly concerned with it being indexed by Google, et al., then you could use a robots.txt file. That said, if you tell robots not to index it, you're also tipping your hand to its existence. ("Ignore the man behind the curtain.")

Two people so far have commented that if you put the directory in robots.txt, it becomes obvious that there's something there. Which is what I devoted half of that first paragraph to. But maybe my idioms are opaque. So, to be clear: Having a file that says what not to look at can lead disreputable entities to look there.

Actually, as Cristian Ciupitu says in a comment:

But if he stores it in a subdirectory, the "curtain" won't be visible. He could use for example public_html/nobots/secret23/index.html.

What this means is that you can combine the use of robots.txt with the other suggestion of not having any links to it. You can also add some sort of authentication scheme.

Improve this answer
5
  • 3
    But if he stores it in a subdirectory, the "curtain" won't be visible. He could use for example public_html/nobots/secret23/index.html.
    – Cristian Ciupitu
    Commented Oct 12, 2009 at 1:42
  • Putting a robots.txt files will notify less reputable search engines that the directory exists and increase the chance it will be found.
    – Donald Byrd
    Commented Oct 12, 2009 at 12:37
  • @ongle: Yes, as stated in the answer.
    – wfaulk
    Commented Oct 12, 2009 at 16:50
  • @wfaulk: I added it only because to many engines the robot.txt is a source of content. Not google, but there are tons of others. Sorry to offend.
    – Donald Byrd
    Commented Oct 12, 2009 at 17:38
  • It should be noted that by listing it in robots.txt, you are essentially marking it as something of interest to anyone investigating your site
    – Matt Simmons
    Commented Oct 12, 2009 at 20:17
8

It is unlikely that it will be found, but you can't trust it. If you don't want the world to see it, put some kind of authentication on it so your friend has to at least log in. Even then, once you share it with your friend you can't be sure he hasn't leaked the secret to someone else. If you must do this, take it down once you no longer need to share it.

Improve this answer
2
  • 1
    So, if I really give it a secure name, like public_html/secret2323ew33443AA321F1 then the probability that they will find it equals the probability of them randomly generating that URL and finding content there, right? (Assume my friend will not leak the URL.) So, there is no way that Google can list the whole recursive content of my public_html, or is there?
    – user9474
    Commented Oct 11, 2009 at 23:21
  • A random name will make it harder to find by dictionary scan. But putting a username/password on it would still be recommended. You always have the option of changing the password if you find your friend is different notions about security than you. But if it is on the Internet, you can't make statements like "no way" -- there is often a way if someone cares enough.
    – Donald Byrd
    Commented Oct 12, 2009 at 12:40
3

If

  1. If the folder above, i.e. ./~dehmann does not have automatic index generation turned on and
  2. there are no public HTML links or the like that reveal the URL of your secret folder

then no, search engines and random visitors will not find the so-called secret folder.

However, most people agree to the axiom that "Security through obscurity doesn't work". I think you'd be better off learning which webserver is used, and taking the time to set up a basic authentication mechanism. It's typically not hard at all -- your ISP's helpdesk could help, or this site...

I think the answer is yes, since Google also uses this technique in Picasa and Google Documents, to publish documents, where the user doesn't want search engines to see it.

Hmn, I don't know what you refer to, but I don't think/hope so. What looks like a very long random directory name is often actually a hash of secret values known to both sides, i.e. there may be some one-way encryption of pre-shared information used for generating that long URL.

Improve this answer
2

It's unlikely that it'll be found, the best way would just be to restrict access in some way, either by host or user/password combination.

Improve this answer
1

If there's neve a reference to it a web crawler likely won't find it. However, it you have the option, still put security on that path, such that your friend must typei in a username/password. I used to do that to the admin directory back in the day when I ran forums in phpBB.

Improve this answer
1

This is a crazy idea, but how about you configure your webserver to only allow authenticated users access to the file. Then give your friend access.

Improve this answer
0

Keep in mind, if you have any links to other websites on your secret page, the target sites' logs would show your secret site as the referring URL. In my experience it seems if you link to google... google will notice your referring URL and attempt to crawl your site.

However, the main question here is: how sensitive is your data?For example, if you have any kind of personal data, e-mail addresses, SSNs, etc, you should definitely put some sort of authentication on it.

Improve this answer
0

If your friends happens to use the Alexa toolbar (by Amazon) or any of a thousand other toolbars or browser helpers/plugins that run within the browser, that toolbar/plugin will know you are visiting the page and can easily send that URL anywhere it wants. Whether any of them do this or not, I don't know, but I doubt that none of them do this.

Improve this answer
0

How much information are you trying to share this way? A better bet may be to send it via encrypted email, assuming that this is possible in your case.

Improve this answer
0

if all you want is to keep search engine from indexing it, just put META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW in your html head section.

robots.txt also works but as said already, this way actually reveal your site to human when they directly access your robots.txt

http://www.robotstxt.org/meta.html

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .