I need to analyze data stored in multiple databases. Each of them contains a few TB of logs and sensor data.

If I use Splunk or ElasticSearch/Kibana, I see 2 solutions:

  1. Batch import everything

  2. Write some scripts to import only the data I need, on demand

This is not optimal because with 1. I duplicate TB of data, and with 2. I need some custom logic that won't be integrated with my analytics tool.

My questions are:

  • Is there a way to perform analytics in place with Splunk or Kibana, i.e. skip the importing step and just read everything from the existing databases?

  • Or is there a tool that would automatically fetch the relevant data and only the relevant data?

  • Can you just index one of the databases and not the duplicate? – Raffi Khatchadourian Aug 5 '14 at 18:04

To be able to search you data quickly i advice Elasticsearch Depending on what the logic behind relevant data should be you can use ElasticSearch Rivers http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/

This allows Elastic search to pull all (relevant) data. My experience says the TB of data you have stored will be reduced to several GB in ElasticSearch

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.