We've been asked by a client to remove a number of pages from being shown up in their public website search results page. I've been into the SSP and created Crawl Rules to remove these pages. All seemed to have worked ok but we have an issue in that landing pages are still showing up in their "www.domain.com/sitearea/" form but not in their "www.domain.com/sitearea/pages/default.aspx".

For each of this type of page we have created one rule to "Exclude" the "aspx" path and another rule to include the "/" path but to "Follow links on the URL without crawling the URL itself". We tried adding rules to exclude the "/" format but that only resulted in all results underneath that being excluded.

Does anybody know how to remove the "area/pages/default.aspx" and the "area/" pats from Search Results?

I'm not sure if it's the "done thing" to ask 2 questions in one but this is in a similar vein so it should be ok. I was wondering if anyone knew of a tool (or if it is possible) to allow site admins to exclude pages from search results (not via SSP/Crawl Rules). I know they can do it at the site level but I was wondering if anything out there enabled this to be done at the page level through either Page or Site Settings?

  • What does your exclusion rule look like? I'll ask someone on the search team about this – Kevin Davis Jan 12 '10 at 2:22
  • Hi Kevin, sorry for slow reply. If, for example, I want to remove www.site.com/subsite/default.aspx from search results, I would think I would only need to add one crawl rule - to exclude www.site.com/subsite/default.aspx. Having done this www.site.com/subsite still shows up. I then add another rule to exclude www.site.com/subsite (no asterisk) but it removes all content that contains that URL. – j.strugnell Mar 15 '10 at 14:17

I'm not sure I understand -- are we talking about excluding pages from public search engines like Google, or from a internal Sharepoint-specific search function?

Well, in both cases robots.txt should work for excluding webpages from indexing by search engines. I'm no Sharepoint expert, but a quick googling seems to show that Sharepoint Search does obey robots.txt, so this would be my first pick.

Here is the main documentation for the format of robots.txt. This doc from Microsoft seems to describe Sharepoint Search management quite well. It says:

SharePoint Portal Server 2003 and SharePoint Server 2007 automatically obey the restrictions that are contained in the Robots.txt file.

- which I again take to mean that Sharepoint Search will obey a robots.txt file.

If your site is publicly accessible, then you might want to open a Google Webmaster Tools account. They have some nice tools troubleshooting various crawling issues, and seeing how your robots.txt will work for your site.


If you want to exclude all pages like "area/pages/default.aspx" and the "area/", the crawl exclusion rule would need to have an asterick at the end i.e. www.domain.com/sitearea/*


I'm having a similar type problem, the indexer is including navigation items in the search, but only for STS_Web objects... or they should be STS_Web objects... the publishing WCM features in SharePoint seem to mark the welcome pages as STS_ListItem_850... not STS_Web... the URL in the search results is the Web URL not the page url (excludes Pages/Default.aspx) it's indexing the welcome page as the Web not the Web itself.


Try creating a search scope with rules based on the contentclass property. All sites use the sts_site value and webs sts_web. Removing any contentclass with those values should prevent the site/ version of homepages from appearing.

  • Hi there, the "/" version has a contentclass of "sts_listitem_850" the same as the full url. – j.strugnell Mar 15 '10 at 14:18

I have added a URL is search result removal - so as to not show in search result page. Not the requirement has changed and those results are to be shown again.. how to get it back..

  • The search result removal adds a crawl rule entry. You need to delete this. – j.strugnell Mar 15 '10 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.