Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Been searching like crazy for hours.

What happened:

While googling my site to check for certain keywords, I used their site preview function (where you get that little popup with an image representation of the page) that shows Google's cached version of pages on my site.

I always assumed that Google would only be able to see site content available to anonymous users. Am I naive?

To my surprise, this cached version showed content that is only seen in the webbrowser when a site user is logged into my site. The URL for the page in question is the same, only the content changes depending on who is logged in / if the user is anonymous. What is even more troublesome is that it is information that is only visible when a staff user is logged in.

I'm using Django/ModWSGI at the application level. I triple-checked that an anonymous user would not be able to see said content.

Now, unless Google has access to login information of a staff user of my site and using that informtation while crawling my site (which I find hard to believe), the cause should be somewhere on the client side.

Question:

Are there webbrowser plugins that push viewed website content to Google for some reason? Of course I'd like to ask my staff users in that case, but this will be more fruitful knowing what to look for. Clients used are Firefox 3.x on Win/Mac/Ubuntu, IE 7/8, Safari on Win/Mac.

How can I defend my site in this respect to avoid such information leakage?

Thanks!

Edit

Further observations / information:

I can google my site using keywords that are normally only visible to certain staff users, thus I conclude that Google must have an association of that keyword with my site stored in its database somewhere (at least I am not sure how else it would work).

The webcache version of said page on my site is about a week old (according to Google) and when I view this cached version, the searched keywords are highlighted, even after the webbrowser cleared its own cache and while I am currently not logged into my site.

share|improve this question

3 Answers 3

Google does only have anonymous access to any site it crawls.

Your CMS of choice may send different content to Google, but this ought to be configurable. Also bear in mind that Google may have crawled the site at a point the content was public, and if it was subsequently made private they may not have updated their index.

I highly doubt there are any browser plugins that send website content to Google for just this reason. There are no end of confidential web pages protected by a user login, and Google would be in an awful lot of hot water if it kept this information, let alone displayed it as search results.

share|improve this answer
    
+1 for correctness and new pointers. Thanks! –  Geradeausanwalt Mar 14 '11 at 10:11

Empty your browser's cache and then try the search or try the search on a machine that has NEVER logged into your site and I'm willing to bet the confidential information won't be there in the search results.

I think it's simply cached information (on your local pc) that you're seeing, especially in light of the fact that the URL can be seen with anonymous info and "content" on the page changes with being logged in.

share|improve this answer
    
Nope, it seems the webbrowser cache has nothing to do with the search results listing after all. –  Geradeausanwalt Mar 14 '11 at 10:10
up vote 0 down vote accepted

Some new facts surfaced that led to solving the case.
Maybe it will help someone else sometime.

1) Google actually has said page in its cache the way I can view it in Google. Since crawling happens only as an anonymous user, the info leakage must happen server-side after all.

2) I narrowed down the info leakage to one certain page (a mean a Django view for all you Djangonauts) that can be accessed via different URLs that only have the username as a difference.

3) One of the other developers messed up by overwriting a server-side variable for that certain page. The variable that usually holds the user that was visiting the page got overwritten by the user whose page one was visiting. If you anonymously visited a staff user's page, you were able to see info that you shouldn't. :-(

share|improve this answer
    
Sounds like you have serious coding problems that allow a rather trivial bypass of authentication, or at least trivial enough that a google request for a page worked without any access credentials. Whilst the #1 recommendation is to go fix the code, you may also want to set a robots.txt that stops crawlers from indexing sensitive pages. It's not a security control at all, but at least avoids a crawl while you're dealing with this security problem. –  Simon at LabSlice-com Mar 14 '11 at 21:16
    
Oh the fix was done within a minute, it was the narrowing it down that was time-consuming. That page itself isn't sensitive, it's certain info in a sidebar. The sidebar exists throughout the site, but the info was only visible to Google at a certain page (the one with the coding problem). –  Geradeausanwalt Mar 15 '11 at 22:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.