Maybe this is a stupid question, but...
I am working with this company and they said they needed to get "permission" to crawl other people's sites. They have a Google Search Appliance And some Google Minis and want to point them at other sites to aggregate content. The end result will be something like a targeted search engine. (All the indexed sites relate to a specific topic)
The only thing they will be doing is:
- Indexing Content from the other sites/domains
- Providing search functionality on their own site that searches the indexed content (like Google, displaying summaries and not the full content)
- The search results will provide links back to the original content
Their intent is not malicious in nature, and is to provide a single site/resource for people to reference on their given topic.
Is there anything illegal or fishy about this process?